Batch request to multiple users fails if one user revokes access

StackOverflow https://stackoverflow.com/questions/21150275

سؤال

I am working on a glassware project using the mirror api.

My app is similar to a news app that delivers article timeline cards to all subscribed users when the article get published. As my user base grows, this will become a problem because, as the app stands now, it is making one API call for every user for every article card. Depending on the user's settings, we could deliver up to 50 articles a day. Given that Google only had a courtesy limit of 1000 Mirror API calls per day, I have already hit that with only 20 users. I am aware that this will be increased when my app gets approved, but I would still like to optimize my code.

Question 1: Am I correct in assuming that implementing batching will decrease the number of API calls I could not find that answered here: https://developers.google.com/glass/batch Say I make a single batch call to post a single timeline item to 1000 of my user's glass timeline, is that counted as 1 Mirror API request, or 1000 Mirror API requests? If the latter, (Question 1.1) am I correct in assuming the only benefit of batching would be to reduce the amount of traffic on my app?

The problem I am running into is that if a user revokes access to my app from (https://security.google.com/settings/security/permissions) when the batch runs, it raises an exception and fails. It ends up logging the user (who still has permissions) out and they are forced to re-authenticate. From my testing, I don't believe it fails the entire batch, so I am unsure if it delivers the timeline card to ALL other users who have not removed access.

Currently, without batching, I am able to catch this error and remove the user from my database if they revoke access. Question 2: How do I catch this error in my code and determine what user it corresponds to. Sample code is below:

def _insert_item_all_users(self):
    """Insert a timeline item to all authorized users."""
    logging.info('Inserting timeline item to all users')
    users = Credentials.all()
    total_users = users.count()

    if total_users > 10:
        return 'Total user count is %d. Aborting broadcast to save your quota' % (
                total_users)
    body = {
            'text': 'Hello Everyone!',
            'notification': {'level': 'DEFAULT'}
    }

    batch_responses = _BatchCallback()
    batch = BatchHttpRequest(callback = batch_responses.callback)
    for user in users:
        creds = StorageByKeyName(
                Credentials, user.key().name(), 'credentials').get()
        mirror_service = util.create_service('mirror', 'v1', creds)
        batch.add(
                mirror_service.timeline().insert(body = body),
                request_id = user.key().name())

    batch.execute(httplib2.Http())
    return 'Successfully sent cards to %d users (%d failed).' % (
            batch_responses.success, batch_responses.failure)

The error I am getting is this:

INFO         2014-01-15 22:42:06,031 client.py:699] Failed to retrieve access token: {
    "error" : "invalid_grant"
}

The access token refresh failed and it raised a oauth2client.client.AccessTokenRefreshError

Steps to reproduce:

  1. Download, install, and run the Glassware Starter Project for Python https://developers.google.com/glass/develop/mirror/quickstart/python
  2. Open 2 browsers (or 2 chrome private browsing windows) and log in (complete the auth2 auth) with 2 different Google accounts (I will refer to them as account one and account two). You should see a welcome timeline card in both users feeds.
  3. On user account one, click the last button that says "Insert a card to all users". If you refresh both accounts pages, you should see a card inserted saying "Hello Everyone!"
  4. Now in account two's browser, open a new tab and go to https://security.google.com/settings/security/permissions Find the app and click "revoke access"
  5. In account one's browser, click the last button that says "Insert a card to all users". This should now log you out and forward you to the Google oath screen to log you back in. Once you log back in you can see that you did indeed receive the timeline card, but if you then re-authenticate account two, they did not.

Apps Project Log

--Send timeline card to all users (before revoking access)
INFO         2014-01-15 22:41:41,217 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:41:41,217 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:41,217 discovery.py:190] URL being requested: https://www.googleapis.com/discovery/v1/apis/mirror/v1/rest?userIp=127.0.0.1
INFO         2014-01-15 22:41:41,999 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:41:41,999 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:41,999 client.py:680] Refreshing access_token
INFO         2014-01-15 22:41:42,519 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:41:42,519 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:42,521 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:42,521 appengine.py:265] get: Got type <class 'model.Credentials'>
INFO         2014-01-15 22:41:42,523 main_handler.py:275] Inserting timeline item to all users
INFO         2014-01-15 22:41:42,529 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:41:42,530 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:42,532 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:41:42,532 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:42,532 discovery.py:190] URL being requested: https://www.googleapis.com/discovery/v1/apis/mirror/v1/rest?userIp=127.0.0.1
INFO         2014-01-15 22:41:42,945 discovery.py:709] URL being requested: https://www.googleapis.com/mirror/v1/timeline?alt=json
INFO         2014-01-15 22:41:42,946 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:41:42,946 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:42,949 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:41:42,950 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:42,950 discovery.py:190] URL being requested: https://www.googleapis.com/discovery/v1/apis/mirror/v1/rest?userIp=127.0.0.1
INFO         2014-01-15 22:41:43,666 discovery.py:709] URL being requested: https://www.googleapis.com/mirror/v1/timeline?alt=json
WARNING    2014-01-15 22:41:43,666 util.py:125] execute() takes at most 1 positional argument (2 given)
INFO         2014-01-15 22:41:44,641 module.py:617] default: "POST / HTTP/1.1" 302 -
INFO         2014-01-15 22:41:44,648 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:41:44,648 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:44,649 discovery.py:190] URL being requested: https://www.googleapis.com/discovery/v1/apis/mirror/v1/rest?userIp=127.0.0.1
INFO         2014-01-15 22:41:45,092 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:41:45,093 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:45,093 client.py:680] Refreshing access_token
INFO         2014-01-15 22:41:45,841 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:41:45,841 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:45,842 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:41:45,843 appengine.py:265] get: Got type <class 'model.Credentials'>
INFO         2014-01-15 22:41:45,850 discovery.py:709] URL being requested: https://www.googleapis.com/mirror/v1/contacts/python-quick-start?alt=json
WARNING    2014-01-15 22:41:45,852 urlfetch_stub.py:482] Stripped prohibited headers from URLFetch request: ['content-length']
INFO         2014-01-15 22:41:46,472 main_handler.py:93] Unable to find Python Quick Start contact.
INFO         2014-01-15 22:41:46,492 discovery.py:709] URL being requested: https://www.googleapis.com/mirror/v1/timeline?maxResults=3&alt=json
WARNING    2014-01-15 22:41:46,494 urlfetch_stub.py:482] Stripped prohibited headers from URLFetch request: ['content-length']
INFO         2014-01-15 22:41:47,028 discovery.py:709] URL being requested: https://www.googleapis.com/mirror/v1/subscriptions?alt=json
WARNING    2014-01-15 22:41:47,031 urlfetch_stub.py:482] Stripped prohibited headers from URLFetch request: ['content-length']
INFO         2014-01-15 22:41:47,562 module.py:617] default: "GET / HTTP/1.1" 200 8163
INFO         2014-01-15 22:41:47,664 module.py:617] default: "GET /static/bootstrap/css/bootstrap-responsive.min.css HTTP/1.1" 304 -
INFO         2014-01-15 22:41:47,665 module.py:617] default: "GET /static/bootstrap/css/bootstrap.min.css HTTP/1.1" 304 -
INFO         2014-01-15 22:41:47,666 module.py:617] default: "GET /static/main.css HTTP/1.1" 304 -
INFO         2014-01-15 22:41:47,668 module.py:617] default: "GET /static/images/chipotle-tube-640x360.jpg HTTP/1.1" 304 -
INFO         2014-01-15 22:41:47,672 module.py:617] default: "GET /static/bootstrap/js/bootstrap.min.js HTTP/1.1" 304 -




--Send timeline card to all users (after revoking access)
INFO         2014-01-15 22:42:02,892 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:42:02,893 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:02,893 discovery.py:190] URL being requested: https://www.googleapis.com/discovery/v1/apis/mirror/v1/rest?userIp=127.0.0.1
INFO         2014-01-15 22:42:03,278 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:42:03,279 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:03,279 client.py:680] Refreshing access_token
INFO         2014-01-15 22:42:03,829 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:42:03,829 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:03,830 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:03,830 appengine.py:265] get: Got type <class 'model.Credentials'>
INFO         2014-01-15 22:42:03,834 main_handler.py:275] Inserting timeline item to all users
INFO         2014-01-15 22:42:03,844 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:42:03,844 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:03,846 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:42:03,847 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:03,847 discovery.py:190] URL being requested: https://www.googleapis.com/discovery/v1/apis/mirror/v1/rest?userIp=127.0.0.1
INFO         2014-01-15 22:42:04,270 discovery.py:709] URL being requested: https://www.googleapis.com/mirror/v1/timeline?alt=json
INFO         2014-01-15 22:42:04,271 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:42:04,271 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:04,275 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:42:04,276 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:04,276 discovery.py:190] URL being requested: https://www.googleapis.com/discovery/v1/apis/mirror/v1/rest?userIp=127.0.0.1
INFO         2014-01-15 22:42:04,705 discovery.py:709] URL being requested: https://www.googleapis.com/mirror/v1/timeline?alt=json
WARNING    2014-01-15 22:42:04,705 util.py:125] execute() takes at most 1 positional argument (2 given)
INFO         2014-01-15 22:42:05,531 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:42:05,531 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:05,532 client.py:680] Refreshing access_token
INFO         2014-01-15 22:42:06,031 client.py:699] Failed to retrieve access token: {
    "error" : "invalid_grant"
}
INFO         2014-01-15 22:42:06,035 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:42:06,035 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:06,038 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:06,038 appengine.py:265] get: Got type <class 'model.Credentials'>
INFO         2014-01-15 22:42:06,043 appengine.py:276] make: Got type <class 'google.appengine.api.datastore_types.Blob'>
INFO         2014-01-15 22:42:06,043 appengine.py:289] validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
INFO         2014-01-15 22:42:06,056 module.py:617] default: "POST / HTTP/1.1" 302 -
INFO         2014-01-15 22:42:06,066 module.py:617] default: "GET /auth HTTP/1.1" 302 -
هل كانت مفيدة؟

المحلول 2

To answer Question 1: No. This will not reduce the number of API calls at all. If you batch 10 requests together, that will still be 10 hits against the API. All this does is reduce the overhead of creating each API call and lets you send them together as a single request.

(As an aside - if you want a quota increase, you can use the form for submitting your Glassware to request just the quota increase. You don't need to specify the rest - just justify why you need it while still developing.)

نصائح أخرى

Two answers for two questions, gathered from stuff we discussed in comments above.

Q1 - Does batching save API quota?

Nope. An API request within a batch still counts as a whole API request.

Batching is for performance. It reduces the number of sockets you have to open and saves you bandwidth, especially if you use gzip and are sending the same payload to many users.

If you'd like more quota, please fill out the Glassware submission form. You can use this even if you just need more quota to continue development.

Q2 - Why is an expired token breaking my whole batch requests?

As you discovered, it was a bug in the Python API Client Library. It should be calling your callback even for failed requests. You patched it like so: 

--- /google-api-python-client-1ab344e0a34d/apiclient/http.py
+++ /google-api-python-client-patched/apiclient/http.py
@@ -50,7 +50,7 @@
 from model import JsonModel
 from oauth2client import util
 from oauth2client.anyjson import simplejson
-
+from oauth2client.client import AccessTokenRefreshError

 DEFAULT_CHUNK_SIZE = 512*1024

@@ -1299,10 +1299,13 @@
     for request_id in self._order:
       resp, content = self._responses[request_id]
       if resp['status'] == '401':
-        redo_order.append(request_id)
-        request = self._requests[request_id]
-        self._refresh_and_apply_credentials(request, http)
-        redo_requests[request_id] = request
+        try:
+          request = self._requests[request_id]
+          self._refresh_and_apply_credentials(request, http)
+          redo_order.append(request_id)
+          redo_requests[request_id] = request
+        except AccessTokenRefreshError:
+          pass

     if redo_requests:
       self._execute(http, redo_order, redo_requests)
مرخصة بموجب: CC-BY-SA مع الإسناد
لا تنتمي إلى StackOverflow
scroll top