Get header values of reply using pycurl

Question

I would like to know some ways to capture and access the header information of the reply when making a request with PyCurl:

c = pycurl.Curl() 
c.setopt(c.URL,'MY_URL')
c.setopt(c.COOKIEFILE,'cookies')
c.setopt(c.COOKIE,'cookies')
c.setopt(c.POST,1)
c.setopt(c.POSTFIELDS,'MY AUTH VALUES')
c.setopt(c.VERBOSE, True)
b = StringIO.StringIO()
c.setopt(c.WRITEFUNCTION, b.write)
c.perform()

The reply will be well-formatted JSON written to buffer b.

I wish to recover the value of the "Location" header in the reply.

When trying to use curl, this value can be seen in the verbose output:

[... Curl output ...]
> GET XXXXXXXXX
[... Request ...]
[... Curl output ...]
< HTTP/1.1 302 Found
[... Other headers ...]
< Location: YYYYYYYYYYYYYYY
[... Rest of reply ...]

How do I recover the value of the Location header from python?

Answer 1

Essentially, a lot of custom functions and registering callback functions. Let's go through curl's verbose output piecewise. First, the bit about the connection can mostly be filled in if you provide your own CURLOPT_OPENSOCKETFUNCTION.

Next, the request headers can are things that you know ahead of time and can print out however you like. For the progress bar, there's CURLOPT_PROGRESSFUNCTION, which allows you to register a callback for updates to the progress "roughly once per second."

You can also register a response header write function (CURLOPT_HEADERFUNCTION) which you can then use to capture and/or display the response headers.

Alternatively, you can use CURLOPT_DEBUGFUNCTION to register callbacks to get information for the headers you send out, get in response, etc.

Answer 2

If you have to use PyCurl

Then you can pass a callback function to get the header information:

# code...

# Callback function invoked when header data is ready
def header(buf):
    # Print header data to stderr
    import sys
    sys.stderr.write(buf)
    # Returning None implies that all bytes were written

# more code...

c.setopt(pycurl.HEADERFUNCTION, header)

# yet more code...

Find out more from the docs.

You can also use requests instead of pycurl

While this may not be possible, and does not directly answer your question, I would recommend that you use the requests library instead of pyCurl:

import requests

payload = {"key":"value"}
cookies = {"key":"value"}

r = requests.post('https://my.example.com', data=payload, cookies=cookies)

location = r.headers["Location"]
content  = r.text

print(content)

It will make your life much easier. Find out more by reading the docs

Answer 3

import pycurl
import cStringIO

buf = cStringIO.StringIO()
URL = 'http://stackoverflow.com/questions/15641080/get-header-values-of-reply-using-pycurl'
c = pycurl.Curl()
c.setopt(c.URL, URL)
c.setopt(c.NOBODY, 1)
c.setopt(c.HEADERFUNCTION, buf.write)
c.perform()

header = buf.getvalue()
print header