python urllib2 request is prepending newline chars to post data

Question

I have a simple python (2.7.5) script using urllib2 to post data to a django (1.6.2) app on a server. When the request arrives at the server, one of the posted values has been truncated and one of the field names has a CR/LF prefix added to it. This seems like an encoding issue but I don't know how to fix it.

The python script posts to the server like this:

auth = 'Basic ' + base64.encodestring('%s:%s' % (user, pwd))
header = {'Authorization': auth }

data = {
    'status': 'TEST',
    'activity_code': 'TEST',
    'sw_version': 'TEST',
    'timestamp': datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
}
data = urllib.urlencode(data)

request = urllib2.Request(url, data, header)
response = urllib2.urlopen(request)
result = json.loads(response.read())
print 'result = ', str(result), '\n'

On the server, there's a django app that processes the request like this:

data = request.POST.copy()
error = {}

try:
    required = ['activity_code', 'timestamp', 'sw_version', 'status']
    for field in required:
        assert len(data.get(field)) > 0
except:
    error['message'] = 'required field is missing (' + field + ')'
    error['debug'] = data

if error:
    content = json.dumps(error, cls=DjangoJSONEncoder)
    return HttpResponse(content, content_type='application/json')

Here's the output of the script after it receives a response from the server:

result = {u'debug': {u'timestamp': u'2014-04-16 14:22:15', u'\r\nstatus': u'TEST', u'activity_code': u'TEST', u'sw_version': u'TE'}, u'message': u'required field is missing (status)'}

Notice that the status field name somehow got prefixed with CR/LF and the sw_version value was truncated from 'TEST' to 'TE'.

Why is this happening??

Answer 1

You did not print the value of your header variable, but the problem you describe could occur if a header value ends with \r\n. Here's why:

A HTTP request header is expected to end with two \r\n (CRLFs), after which the POST data comes. The urllib2 module will not validate header values to check if you have a CRLF in your value. If you do, the server thinks that the header ends one CRLF "too early". This will cause the POST data to start with the last CRLF of your actual header end marker. This, in turn will make the content-length header (counting the length of your original post data) to truncate the post data by two characters.

So I would recommend double-checking your headers.