It is probably easiest to do a custom mapping function, which keeps things simple and readable:
CreateMap<Contact, Destination>().ConvertUsing(c => MapContactToDestination(c));
Destination MapContactToDestination(Contact c)
{
//logic here for handling conversion
}