It looks like a service account for offline access was what I'm looking for.
Inspired by this thread: Trouble with Google Apps API and Service Accounts in Ruby
I found out that I can create a service account on the apps marketplace configuration. Project -> APIs & Auth -> Credentials -> Create New Client Id -> Choose Service account. And in my code use Signet to provide domain level authorization for the end-users:
key = Google::APIClient::PKCS12.load_key('privatekey.p12', 'notasecret')
client = Google::APIClient.new
client.authorization = Signet::OAuth2::Client.new(
:token_credential_uri => 'https://accounts.google.com/o/oauth2/token',
:audience => 'https://accounts.google.com/o/oauth2/token',
:scope => 'https://www.googleapis.com/auth/calendar',
:issuer => '<email-address-of-service-account>@developer.gserviceaccount.com',
:signing_key => key,
:person => email)
client.authorization.fetch_access_token!
Not only does this allowed me to authenticate without prompting the end-user at any phase during the process - but I don't require the refresh_token anymore, and I can execute the API calls with domain level authorizations. The person option allows me to specify which end-user I'm using the API call for.