Python CookieJar saves cookie, but doesn't send it to website

Question

I am trying to login to website using urllib2 and cookiejar. It saves the session id, but when I try to open another link, which requires authentication it says that I am not logged in. What am I doing wrong?

Here's the code, which fails for me:

import urllib
import urllib2
import cookielib

cookieJar = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookieJar))

# Gives response saying that I logged in succesfully
response = opener.open("http://site.com/login", "username=testuser&password=" + md5encode("testpassword"))

# Gives response saying that I am not logged in
response1 = opener.open("http://site.com/check")

Answer 1

Your implementation seems fine... and should work.

It should be sending in the correct cookies, but I see it as the case when the site is actually not logging you in.

How can you say that its not sending the cookies or may be cookies that you are getting are not the one that authenticates you.

Use : response.info() to see the headers of the responses to see what cookies you are receiving actually.

The site may not be logging you in because :

• Its having a check on User-agent that you are not setting, since some sites open from 4 major browsers only to disallow bot access.

• The site might be looking for some special hidden form field that you might not be sending in.

1 piece of advise:

from urllib import urlencode
# Use urlencode to encode your data

data = urlencode(dict(username='testuser', password=md5encode("testpassword")))
response = opener.open("http://site.com/login", data)

Moreover 1 thing is strange here :

• You are md5 encoding your password before sending it over. (Strange)
• This is generally done by the server before comparing to database.
• This is possible only if the site.com implements md5 in javascript.
• Its a very rare case, since only may be 0.01 % websites do that..
• Check that - that might be the problem, and you are providing the hashed form and not the actual password to the server.
• So, server would have been again calculating a md5 for your md5 hash.

Check out.. !! :)

Answer 2

I had a similar problem with my own test server, which worked fine with a browser, but not with the urllib2.build_opener solution.

The problem seems to be in urllib2. As these answers suggest, it's easy to use more powerful mechanize library instead of urllib2:

cookieJar = cookielib.CookieJar()
browser = mechanize.Browser()
browser.set_cookiejar(cookieJar)
opener = mechanize.build_opener(*browser.handlers)

And the opener will work as expected!