Python - BytesIO and urllib2.add_data

Question

I have a BytesIO that I'm adding various bytes to. I want to send this in a urllib2.Request via the request.add_data method. How do I do this? When I try

# create request ....
bytesio = BytesIO()
bytesio.write(open("C:\img.jpg", "rb").read())
request.add_data(bytesio.getvalue()) 
bytesio.close()

urllib2.urlopen(request) # error "expected buffer, got bytes"

What am I doing wrong? I'm new to Python and not sure how to create a buffer from a BytesIO. Also, when I just try:

request.add_data(bytesio)  # instead of bytesio.getvalue()

I get a "I/O operation on closed file". If I try to wait until after urlopen to call bytesio.close, then the request just hangs because it's waiting for bytesio to be closed.

What do I need to do?

Answer

request.add_data(str(btyesio.getvalue()))
bytesio.close()

Casting to a string made it happy. I haven't tried to see if it all works with StringIO and I haven't tried the differences between Python 2.x and 3.x.

Answer 1

Simplest solution: don't use a BytesIO, you don't need it.

urllib2.Request.add_data expects it's argument to be a string, so just give it one.

the call:

bytesio.write(open("C:\img.jpg", "rb").read())

reads the whole file into memory, then writes it to bytesios memory. That means you already have the string in mory, you don't need it twice. So just try:

request = urllib2.Request('http://www.site.com')
with open("C:\img.jpg", "rb") as inputfile:
    request.add_data(inputfile.read())
urllib2.urlopen(request)

Answer 2

Just remove this line(add it at the end):

bytesio.close()

The other code seems to work for me:

bytesio = BytesIO()
bytesio.write(open("C:\img.jpg", "rb").read())
request = urllib2.Request('http://www.site.com')
request.add_data(bytesio.getvalue()) 

urllib2.urlopen(request) # error "expected buffer, got bytes"
bytesio.close()

>>In [30]: urllib2.urlopen(request)
Out[30]: <addinfourl at 52264040 whose fp = <socket._fileobject object at 0x315a450>>

Or using StringIO:

sio = StringIO.StringIO(open("C:\img.jpg", "rb").read())
request = urllib2.Request('http://www.site.com')
request.add_data(sio.getvalue()) 

urllib2.urlopen(request) # error "expected buffer, got bytes"
sio.close()

>>In [14]: urllib2.urlopen(request)
Out[14]: <addinfourl at 49067360 whose fp = <socket._fileobject object at 0x2dfb3d0>>

Answer 3

There is no BytesIO.getvalue() method because it's not needed. Instead, just keep a reference to the underlying buffer.

This works with lists and arrays, as well as bytes objects, but it's sort of a coincidence, rather than an actual design goal...

Answer 4

You can get the whole data of a BytesIO with:

esio.seek(0, os.SEEK_SET)
esio.read()

(SEEK_SET isn't needed, but I use it to clarify things)

So your example:

# create request ....
bytesio = BytesIO()
bytesio.write(open("C:\img.jpg", "rb").read())
bytesio.seek(0, os.SEEK_SET)
request.add_data(bytesio.read()) 
bytesio.close()

urllib2.urlopen(request)