How can I see the actual data Requests sends over the wire?

Question

(This is a follow-up question to this one)

How can I tell urllib3 to log the FULL request, including, but not limited to:

• URL
• query parameters
• headers
• body
• and anything else which is sent inside the request (I am not sure there is anything else, but if there is something else, I also want to see it)

I am having trouble connecting to LinkedIn with OAuth (a similar implementation works with Google and Facebook), and I would like to see exactly what requests are being sent. I suspect the auth_token is not being provided, but I need to confirm this. For that, I need urllib3 to show the full requests, since they are over HTTPS and I can not analyze network traffic to see them (end-to-end encryption).

Answer 1

You can access the PreparedRequest object that was actually sent after the fact of a request by using the request accessor, e.g. print dir(r.request).

Answer 2

You can hack into the requests hooks system instead, then use that to track responses and their requests:

from requests import hooks


_orig_default_hooks = hooks.default_hooks


def my_default_hooks():
    hooks = _orig_default_hooks()
    hooks['response'].append(response_hook)
    return hooks


hooks.default_hooks = my_default_hooks


# requests.models is imported by the requests package, so we need to ensure it's reference
# to default_hooks is updated too.
import requests.models
requests.models.default_hooks = my_default_hooks


def response_hook(r, **kw):
    req = r.request
    print req.headers

The reqeuests.hooks.default_hooks() function is called for each and every request created, and by injecting your own response event hook you get to see each and every response received. Responses have a .request attribute, which is a PreparedRequest instance, on which you'll find the .headers and .body attributes for your inspection.