You are correct that a service account won't work with the Contacts API, however you don't have to authorize it every time you use it. You can provide domain-wide delegation with two-legged OAuth 1.0a instead of 2.0. See https://developers.google.com/google-apps/contacts/v3/index#authorizing_requests_to_the_+wzxhzdk8+_service and https://developers.google.com/accounts/docs/OAuth#GoogleAppsOAuth.
Depending on your setup, you can also use the three-legged OAuth 2.0 flow for a single administrator account and save the refresh token. This would let you re-use that authorization without requiring explicit permission again.