Why is my HTTP POST request data string (probably) being incorrectly encoded?

Question

I'm having some trouble with twisted.web.client.Agent...

I think the string data in my post request isn't being formatted properly. I'm trying to do something analogous to this synchronous code:

from urllib import urlencode
import urllib2

page = 'http://example.com/'
id_string = 'this:is,my:id:string'

req = urllib2.Request(page, data=urlencode({'id': id_string}))  # urlencode call returns 'id=this%3Ais%2Cmy%3Aid%3Astring'
resp = urllib2.urlopen(req)

Here's how I'm building my Agent request as of right now:

from urllib import urlencode
from StringIO import StringIO

page = 'http://example.com/'
id_string = 'my:id_string'

head = {'User-Agent': ['user agent goes here']}
data = urlencode({'id': id_string})

request = agent.request('POST', page, Headers(head), FileBodyProducer(StringIO(data)))
request.addCallback(foo)

Because of the HTTP response I'm getting (null JSON string), I'm beginning to suspect that the id is not being properly encoded in the POST request, but I'm not sure what I can be doing about it. Is using urlencode with the Agent.request call valid? Is there another way I should be encoding these things?

EDIT: Some kind IRC guys have suggested that the problem may stem from the fact that I didn't send the header information that indicates the data is encoded in a url string. I know remarkably little about this kind of stuff... Can anybody point me in the right direction?

Answer 1

As requested, here's my comment in the form of an answer:

HTTP requests with bodies should have the Content-Type header set (to tell the server how to interpret the bytes in the body); in this case, it seems the server is expecting URL-encoded data, like a web browser would send when a form is filled out.

urllib2.Request apparently defaults the content type for you, but the twisted library seems to need it to be set manually. In this case, you want a content type of application/x-www-form-urlencoded.