The Address Book cannot be accessed without permission. No third-party API can get in, because internally, these API's need to go through the same permission checks as you need to. No app can get into a user's address book without the user's permission.
This is because of a security issue that Path, and some other apps, uploaded its users' address books to their own servers to use for whatever reason. To read more about it, look here
After this surfaced, Apple required the user's permission to access the user's contacts. Apple's iOS platform is possibly the most secure operating systems today, and there are few security holes that exist in their API's (minus the goto fail;
mess-up).
App Store reviewed apps are more secure for the user than Ad-Hoc apps. The developers at Apple make sure that you do not do anything malicious with the user's contacts. In Ad-Hoc apps, there is no checkup. So, if you wanted to do anything dirty with their contacts in an Ad-Hoc app, you technically could (if the user gives you permission at all). You do not need to state what you will be doing with the permission, and so you are able to take advantage of the user's trust in you.
If you want the company to trust the app, suggest that they look it over with their own reviewers. If they don't think you are doing anything fishy, you are good.