There is no way to add Contacts folders from mailboxes shared with you.

It works with Public Folder contacts and SharePoint Lists. You can see the list of Address Books when clicking To, From, or CC buttons when composing a message. It can be useful to have Public Folder or SharePoint Contacts to choose from.

To add Contacts folder to the list of Address Books, you need to do the following: