Python authentication with requests. Why it doesn't work? .NET and wikipedia

Question

It has been few days that I tried to scrape this page: http://londoncoffeeguide.com/

I tried to use requests or scrapy, but I'm newby to the scrapin world and I cannot find a way to login. Is it possible to login to this website with requests and use BeautifulSoup to scrape it? Or is it possible to do it with scrapy?

Furthermore, I tried to test requests following this example, and to test it on wikipedia, using the same pages linked there I tried this:

import requests
from bs4 import BeautifulSoup as bs


def get_login_token(raw_resp):
    soup = bs(raw_resp.text, 'lxml')
    token = [n['value'] for n in soup.find_all('input')
    if n['name'] == 'wpLoginToken']
    return token[0]


payload = {
    'wpName': 'my_login',
    'wpPassword': 'my_pass!',
    'wpLoginAttempt': 'Log in',
    #'wpLoginToken': '',
    }

with requests.session() as s:
    resp = s.get('http://en.wikipedia.org/w/index.php?title=Special:UserLogin')
    payload['wpLoginToken'] = get_login_token(resp)
    print payload
    response_post = s.post('http://en.wikipedia.org/w/index.php?title=Special:UserLogin&action=submitlogin&type=login', data=payload)
    response = s.get('http://en.wikipedia.org/wiki/Special:Watchlist')

    r = bs(response.content)
    print r.get_text()

What I see is that I still get the suggestion to login in order to see the wishlist page.

Where is the mistake?

Answer 1

I got this to login (yes i created an account and tested it)

from mechanize import Browser
    br = Browser()
    br.set_handle_robots(False)
    br.addheaders = [('User-agent', 'Firefox')]
    br.open("http://www.londoncoffeeguide.com")
    for form in br.forms():
        if form.attrs['id'] == 'form':
            br.form = form
    br.form['p$lt$zoneContent$PagePlaceholder$p$lt$zoneRight$logonform$Login1$UserName'] = 'username goes here'
    br.form['p$lt$zoneContent$PagePlaceholder$p$lt$zoneRight$logonform$Login1$Password'] = 'password goes here'
    response = br.submit()

then you can pass response.read() to beautiful soup and do all kinds of stuff