Upload images and metadata to public Amazon S3 bucket with Python requests

Question

I know, there's the boto library for Python, however, all I'd like to do is uploading a lot of image files including metadata to a public S3 bucket. The images should go into various sub-directories inside the bucket.

With cURL, this is supposed to be working:

curl -v -F "key=test/test.jpg" -F "file=@test.jpg" http://my-public-bucket.s3.amazonaws.com/

So I figure that should be doable with urllib, urllib2 and/or Python requests only. But how? I'm totally new to Amazon S3 ... and cURL.

Also what's the best way for storing some meta data along with the images? An additional JSON-string file?

Answer 1

Your cURL string translates into roughly the following:

import requests

url = 'http://my-public-bucket.s3.amazonaws.com/'
files = {
    'key': ('', 'test/test.jpg'),
    'file': open('test.jpg', 'rb'),
}

r = requests.post(url, files=files)

The general form of Requests' multipart upload syntax is found in this StackOverflow answer.

Answer 2

Using boto (version 2.6.0) you'd do it like this:

import boto

connection = boto.connect_s3()
bucket = connection.get_bucket('mybucket')
key = bucket.new_key('myimage.jpg')
key.set_contents_from_filename('myimage.jpg')
key.set_metadata(...)

Make sure you've got the credentials in the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

That's it.

Answer 3

Works with Python Requests only:

import requests
r = requests.post('my_public_bucket', files={'file': open('/path/test.txt', 'rb')}, data={'key': 'test/test.txt'})

Answer 4

To upload to a signed url and requests I had to do this:

with open('photo_1.jpg', 'rb') as content_file:
    content = content_file.read()
result = requests.put(url=upload_url, headers={}, data=content)

This is bad because it loads everything into memory, but it should get you past the initial hump.

Also when using curl I had to use the a different option:

curl -X PUT --upload-file photo_1.jpg <url>

Note: When I created the url at my server with boto I set headers=None so that headers would not be an issue.