Your best friend would be your custom module that has hook_field_access() properly set up in it.
EXAMPLE:
function MYMODULE_field_access ($op, $field, $entity_type, $entity, $account) {
switch ($entity->type) {
case 'REFERENCED_NODE_TYPE_NAME': {
///DO YOUR STUFF HERE: SET A $VAR TO 'TRUE' OR 'FALSE' DEPENDING IF YOU WANT TO GIVE ACCESS.
}
}
return $var;
}