Writing files to Dropbox account from GAE

Question

I am trying to create files in a Dropbox.com folder from a GAE application. I have done all the steps the register a Dropbox application and installed the Python SDK from Dropbox locally on my development machine. (see dropbox.com API). It all works perfectly when I use the cli_client.py test script in the dropbox SDK on my local machine to access dropbox - can 'put' files etc.

I now want to start working in GAE environment, so things get a bit tricky. Some help would be useful.

For those familiar with the Dropbox API code, I had the following issues thus far:

Issue 1

The rest.py Dropbox API module uses pkg_resources to get the certs installed in site-packages of a local machine installation. I replaced

TRUSTED_CERT_FILE = pkg_resources.resource_filename(__name__, 'trusted-certs.crt')

with

TRUSTED_CERT_FILE = file('trusted-certs.crt')

and placed the cert file in my GAE application directory. Perhaps this is not quite right; see my authentication error code below.

Issue 2

The session.py Dropbox API module uses oauth module, so I changed the include to appengine oauth.

But raised an exception that GAE's oauth does not have OAuthConsumer method used by the Dropbox session.py module. So i downloaded oauth 1.0 and added to my application an now import this instead of GAE oauth.

Issue 3

GAE ssl module does not seem to have CERT_REQUIRED property.

This is a constant, so I changed

self.cert_reqs = ssl.CERT_REQUIRED

to

self.cert_reqs = 2

This is used when calling

ssl.wrap_socket(sock, cert_reqs=self.cert_reqs, ca_certs=self.ca_certs)

Authentication Error

But I still can't connect to Dropbox:

Status: 401
Reason: Unauthorized
Body: {"error": "Authentication failed"}
Headers: [('date', 'Sun, 19 Feb 2012 15:11:12 GMT'), ('transfer-encoding', 'chunked'), ('connection', 'keep-alive'), ('content-type', 'application/json'), ('server', 'dbws')]

Answer 1

Here's my patched version of Dropbox Python SDK 1.4 which works well for me with Python 2.7 GAE: dropbox_python_sdk_gae_patched.7z.base64. No extra third-party libraries needed, only those provided by GAE environment.

Only file uploading (put_file) is tested. Here're setup steps:

1. Unpack archive to the root folder of GAE application (if main app is in the root folder). You can decode BASE64 using Base64 Encoder/Decoder: base64.exe -d dropbox_python_sdk_gae_patched.7z.base64 dropbox_python_sdk_gae_patched.7z.
2. Setup APP_KEY, APP_SECRET, ACCESS_TYPE, ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET. First three are configured at dropbox application creation time. Last two are obtained when granting application access to specific dropbox account, you can get them through cli_client.py (from DB Python SDK) from token_store.txt file.

3. Use in the code like this:

   import dropbox
   # ...
   def DropboxUpload(path, data):
       sess = dropbox.session.DropboxSession(APP_KEY, APP_SECRET, ACCESS_TYPE)
       sess.set_token(ACCESS_TOKEN_KEY, ACCESS_TOKEN_SECRET)
       cli = dropbox.client.DropboxClient(sess)
       data_file = StringIO.StringIO(data)
       return cli.put_file(path, data_file)
   # ...
   import json
   class DropboxUploadHandlerExample(webapp2.RequestHandler):
       def get(self):
           url = "http://www.google.com/"
           result = urlfetch.fetch(url)
           self.response.headers['Content-Type'] = 'application/json'
           self.response.out.write(json.dumps(DropboxUpload('/fetch_result.dat', result.content)))
   

Answer 2

I successfully uploaded from Google Appengine to Dropbox with my own patched version of the Dropbox SDK: https://github.com/cklein/dropbox-client-python

The usage of urllib2 was replaced by huTools.http: https://github.com/hudora/huTools/

This is the code that is called in a request handler:

    db_client = dropbox.get_dropbox_client(consumer_key='', consumer_secret='', access_token_key='', access_token_secret='')
    fileobj = StringIO.StringIO(data)
    path = '/some/path/filename'
    resp = db_client.put_file(path, fileobj)
    fileobj.close()

Answer 3

As of April 2016, none of the other suggestions work. (Dropbox API version 2, Python SDK version 6.2).

If you only need a few of the SDK functions, I found it easiest to just use the HTTP API directly:

def files_upload(f, path, mode='add', autorename=False, mute=False):

    args = {
        'path': path,
        'mode': mode,
        'autorename': autorename,
        'mute': mute,
    }

    headers = {
        'Authorization': 'Bearer {}'.format(ACCESS_TOKEN),
        'Dropbox-API-Arg': json.dumps(args),
        'Content-Type': 'application/octet-stream',
    }

    request = urllib2.Request('https://content.dropboxapi.com/2/files/upload', f, headers=headers)
    r = urllib2.urlopen(request)

Answer 4

I have patched the Dropbox Python SDK version 2.2 to work on Google App Engine. Please find the relevant code here:

https://github.com/duncanhawthorne/gae-dropbox-python

The relevant code patch (copied from github) for rest.py is here:

 import io
 import pkg_resources
-import socket
+#import socket
 import ssl
 import sys
 import urllib
+import urllib2

+def mock_urlopen(method,url,body,headers,preload_content):
+    request = urllib2.Request(url, body, headers=headers)
+    r = urllib2.urlopen(request)
+    return r         
+    
 try:
     import json
 except ImportError:
 @@ -23,7 +29,10 @@

 SDK_VERSION = "2.2.0"

-TRUSTED_CERT_FILE = pkg_resources.resource_filename(__name__, 'trusted-certs.crt')
+try:
+    TRUSTED_CERT_FILE = pkg_resources.resource_filename(__name__, 'trusted-certs.crt')
+except:
+    TRUSTED_CERT_FILE = file('trusted-certs.crt')


 class RESTResponse(io.IOBase):
 @@ -125,6 +134,7 @@ def flush(self):
         pass

 def create_connection(address):
+    return
     host, port = address
     err = None
     for res in socket.getaddrinfo(host, port, 0, socket.SOCK_STREAM):
 @@ -152,7 +162,7 @@ def json_loadb(data):


 class RESTClientObject(object):
-    def __init__(self, max_reusable_connections=8, mock_urlopen=None):
+    def __init__(self, max_reusable_connections=8, mock_urlopen=mock_urlopen):
         """
         Parameters
             max_reusable_connections
 @@ -206,7 +216,7 @@ def request(self, method, url, post_params=None, body=None, headers=None, raw_re
                 raise ValueError("headers should not contain newlines (%s: %s)" %
                                  (key, value))

-        try:
+        if True:
             # Grab a connection from the pool to make the request.
             # We return it to the pool when caller close() the response
             urlopen = self.mock_urlopen if self.mock_urlopen else self.pool_manager.urlopen
 @@ -217,14 +227,14 @@ def request(self, method, url, post_params=None, body=None, headers=None, raw_re
                 headers=headers,
                 preload_content=False
             )
-            r = RESTResponse(r) # wrap up the urllib3 response before proceeding
-        except socket.error as e:
-            raise RESTSocketError(url, e)
-        except urllib3.exceptions.SSLError as e:
-            raise RESTSocketError(url, "SSL certificate error: %s" % e)
+            #r = RESTResponse(r) # wrap up the urllib3 response before proceeding
+        #except socket.error as e:
+        #    raise RESTSocketError(url, e)
+        #except urllib3.exceptions.SSLError as e:
+        #    raise RESTSocketError(url, "SSL certificate error: %s" % e)

-        if r.status not in (200, 206):
-            raise ErrorResponse(r, r.read())
+        #if r.status not in (200, 206):
+        #    raise ErrorResponse(r, r.read())

         return self.process_response(r, raw_response)

 @@ -321,10 +331,11 @@ def PUT(cls, *n, **kw):
         return cls.IMPL.PUT(*n, **kw)


-class RESTSocketError(socket.error):
+class RESTSocketError():
     """A light wrapper for ``socket.error`` that adds some more information."""

     def __init__(self, host, e):
+        return
         msg = "Error connecting to \"%s\": %s" % (host, str(e))
         socket.error.__init__(self, msg)