urllib2 - Setting multiple header fields

Question

I'm trying to fetch a site which requieres the correct user-agent and also basic authentification.

  URL = 'http://localhost'                                           
  Creds = base64.encodestring('user:password')                                    
  request = urllib2.Request(url=URL)                                             
  request.addheaders = [('User-Agent', 'Mozilla/6.0 (X11; Linux x86_64; rv:24.0) Gecko/20140205     Firefox/27.0 Iceweasel/25.3.0'), ('Authorization', 'Basic %s' % Creds)]           
  Opener = urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1))            
  Response = Opener.open(request).read()                                      
  print Response

I'm confused 'cause if I just add one header by using request.add_header(foo) it works and I can see the debug output from HTTPHandler. But if I start to set more than one header field I just get back the result without debug information.

Answer 1

>>> import base64
>>> base64.encodestring('user:password')
'dXNlcjpwYXNzd29yZA==\n'

Notice the newline at the end. It might cause the premature end of headers. To fix it, use b64encode() instead:

>>> base64.b64encode('user:password')
'dXNlcjpwYXNzd29yZA=='

Another issue is with request.addheaders, use opener.addheaders or request.add_header() instead:

#/usr/bin/env python
import urllib2
import base64

url = 'http://localhost'
creds = base64.b64encode('user:password')
request = urllib2.Request(url)
request.add_header('Authorization', 'Basic ' + creds)
request.add_header('User-Agent',
                   'Mozilla/6.0 (X11; Linux x86_64; rv:24.0) '
                   'Gecko/20140205     Firefox/27.0 Iceweasel/25.3.0')
opener = urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1))
content = opener.open(request).read()
print content