Simulating HTTP POST request produces unexpected result

Question

---SOLVED---

It turned out the request body had literal r"\n" (repr: "\\n") characters in it, and since I simply copy pasted the body as a Python string, Python thought I was giving it newline characters rather than escaped newline characters.

The reason this causes a Bad Request is as follows: So the body was JSON, and in JSON you have to escape all your newline characters by definition. So when the server loads the JSON object from the raw text, an error is thrown causing Bad Request

I realised this because the Content-Length header was different in both cases (\n is one char while \\\n is two chars, although perhaps the Content-Length doesn't actually matter.

Also it is noteworthy that when a lower Content-Length is sent, Bad Request is also returned. I believe this is because the JSON body gets truncated, and the server doesn't accept the important char (e.g. closing brace or something)

--- Problem:---

Summary:

I am trying to use Python to simulate a POST request to bitbucket.org performed within my Firefox web browser. Here is what I did:

1. Tracked the POST request using Firebug
2. Copied the POST request headers
3. Copied the POST request body (in application/json format)

Code:

Here is the code I use to POST my request, but it's a bit long and not very relevant. My Content-Type is application/json, and my POST body is a JSON-encoded string.

    dataString = '{"branch":"master","files":[{"path":"readme.txt","content":"ntestxx\n \n"}],"message":"readme.txt edited online with Bitbucket","parents":["465305dc4da32f91da057b65297cda9b72c"],"repository":{"full_name":"minesite/ica-i18n"},"timestamp":"2014-03-20T23:49:29.759Z","transient":false}'
    headers = {'X-CSRFToken': '6TqWjCl698U99Iu6ZYGBAloCxZ', 'Content-Length': '2190', 'Accept-Language': 'en,en-us;q=0.7,zh;q=0.3', 'X-NewRelic-ID': 'VwMGVVZSGwIIUFBQDwU=, VwMGVVZSGwIIUFBQDwU=', 'Cookie': 'csrftoken=6TqWjCl698U99Iu6ZYGBAloCxZ; __utma=254090395.1171276563.1394767875.1394776803.1395358874.3; __utmc=254090395; __utmz=254090395.1394776803.2.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); bb_session=gpqergylgoa7icpwosqsbpxig0; __utmv=254090395.|1=isBBUser=true=1; recently-viewed-repos_1701252=3802872%2C108928; __utmb=254090395.21.9.1395359363952', 'Connection': 'keep-alive', 'Accept': 'application/json, text/javascript, */*; q=0.01', 'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0', 'Host': 'bitbucket.org', 'X-Requested-With': 'XMLHttpRequest', 'Pragma': 'no-cache', 'Cache-Control': 'no-cache', 'Referer': 'https://bitbucket.org/xxxxxx/xxxxxxx/src/465305dc4da32f91da057b6529a8e4/readme.txt?at=master', 'Content-Type': 'application/json; charset=UTF-8', 'Accept-Encoding': 'gzip, deflate'}
    edit = requests.post("https://bitbucket.org/!api/internal/repositories/xxxxxxx/xxxxxxxx/oecommits/", data=dataString, headers=headers)

Results vs. expected results:

When I perform the POST request using my Firefox web browser (using Firebug's "resend request" function), I get a 409 CONFLICT response (Which is the desired response! I am simulating a request to an online editor, so that should be the correct response to a re-sent edit).

However, when I try to simulate the request by copying the request header and the request body, I get a 400 BAD REQUEST response, and the response contains no other information, so I don't even know what my problem is.

Regardless of how many times I send the POST in the web-browser (despite an incorrect timestamp), it achieves the intended outcome, but the server refuses to accept any requests I make using the python requests library.

Response using browser request:

Headers

HTTP/1.1 409 CONFLICT
Server: nginx/1.5.10
Date: Fri, 21 Mar 2014 00:20:55 GMT
Content-Type: text/plain
Content-Length: 45
Connection: keep-alive
x-served-by: app16
X-Render-Time: 0.558492183685
Content-Language: en
X-Static-Version: 48695e7c3140
Vary: Authorization, Accept-Language, Cookie
X-Version: e6778a5040f7
Etag: "92f0b780984e984140de0f8ed0a3992c"
X-Frame-Options: SAMEORIGIN
X-Request-Count: 483
X-NewRelic-App-Data: PxQEVFdXCAITVVlWBgMPUkYdFGQHBDcQUQxLA1tMXV1dSn8UXwJHCwtYGAMPF1pGUw8EFhlQRxYXH1dDC0gKDEQHSgxZVBpaUgtdDVQTQFgrWFsICAZ9V1kQIg1aXF4SLFBYVw4DEUxTEF0DTF0WHgNJCU8EVApUUgUHVFFQCgQCU1FXGwMGX1QdFAEBUVVbA1AJVQEBB1FSA11DHQdSDhdTag==

Body

Specified change not on head of branch master

Response using python request:

Headers

content-length: 11
x-served-by: app10
x-render-time: 0.012787103653
content-language: en
content-type: text/plain
vary: Authorization, Accept-Language, Cookie
connection: keep-alive
server: nginx/1.5.10
x-version: e6778a5040f7
etag: "825644f747baab2c00e420dbbc39e4b3"
x-request-count: 321
x-newrelic-app-data: PxQEVFdXCAITVVlWBgMPUkYdFGQHBDcQUQxLA1tMXV1dSn8UXwJHCwtYGAMPF1pGUw8EFhlQRxYXH1dDC0gRB0MNTRBbXQ5gVhZWFEMCVkBIBhtRSFMJAARQUlsDBw9VXAIBC1tWVU4CUwtUFBpVAwFcWgdTVQIAXQBRWQQAGh9WBQ0RUmw=
date: Fri, 21 Mar 2014 00:51:01 GMT
x-frame-options: SAMEORIGIN
x-static-version: 48695e7c3140

Body

Bad Request

Some of my ideas:

I am thinking that perhaps there is another component to a HTTP POST request that I need to simulate? Perhaps when Firefox sends a POST request, there is some header or wrapper added that makes the request valid?

Or is there something more to a POST request than just a method, headers, and body?

Maybe it's something to do with the fact that it's HTTPS instead of HTTP?

Update:

I have tried sending the "sent cookies" in with the request, to little success.

Answer 1

Or is there something more to a POST request than just a method, headers, and body?

No. The important part are the request headers. They should be exactly the same in both cases.

Because Firebug can just track the network requests inside Firefox, you'll need an external network analyzer like Wireshark to track the requests coming from your Python script. Of course you need to run it on the server where the script lies.

Another solution would be to run your request against a local web server and log the request information there.

Then you'll be able to compare the request made in the browser with the one from your script.