I was having issues with this as well. Once I set the addressBook property on the ABPersonViewController, it starting working as I expected.
addressBook Optional. The address book from which to obtain the contact to display. @property(nonatomic, readwrite) ABAddressBookRef addressBook