Failed to screen scrape ASP.Net website while posting data

Question

Getting Invalid postback or callback argument error while trying to screen scrape a website which has build on ASP.NET.

Fist request of landing page has no issue. It's raising exception when I posts form data after changing one of drop-down field value.

"""
Invalid postback or callback argument.  Event validation is enabled using
<pages enableEventValidation="true"/> in configuration or <%@ Page
EnableEventValidation="true" %> in a page.  For security purposes, this feature
verifies that arguments to postback or callback events originate from the server
control that originally rendered them.  If the data is valid and expected, use
the ClientScriptManager.RegisterForEventValidation method in order to register
the postback or callback data for validation. 
"""

Here's my try:

#!/bin/env python
import sys
import requests
from bs4 import BeautifulSoup

HOST = 'forms.toyotabharat.com'
URL = 'http://%s/pricelist-dealer.aspx' % HOST
HEADERS = {
    'Host': HOST,
    'Origin': 'http://%s' % HOST,
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Firefox/38.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'Accept-Encoding': 'gzip, deflate',
    'Connection': 'keep-alive'
}

session = requests.Session()

r = session.get(URL, headers=HEADERS)

if r.status_code != requests.codes.ok:
    sys.exit()

soup = BeautifulSoup(r.content)

# ASP validation and session fields
view_state = soup.select("#__VIEWSTATE")[0]['value']
view_state_generator = soup.select("#__VIEWSTATEGENERATOR")[0]['value']
event_validation = soup.select("#__EVENTVALIDATION")[0]['value']

FORM_FIELDS = {
    '__EVENTTARGET': 'cboState',
    '__EVENTARGUMENT': '',
    '__LASTFOCUS': '',
    '__VIEWSTATE': view_state,
    '__VIEWSTATEGENERATOR': view_state_generator,
    '__EVENTVALIDATION': event_validation,
    'cboState': '3',
    'cboCity': '-1',
    'hdDealerMaps': 'True',
}

# POST form fields
r = session.post(URL, data=FORM_FIELDS, headers=HEADERS, cookies=r.cookies.get_dict())

if r.status_code != requests.codes.ok:
    print "Failed with status_code %d" % r.status_code
    sys.exit()

soup = BeautifulSoup(r.content)

Answer 1

You where basically on the right track. I got it running with a few changes.

Invalid postback or callback argument.

The error message is really helpful. If you read the msdn page of it there is a hint.

Summarized as: don't post parameters or values which are not in the form which you get with GET

In your case means that that if you select a State it should be one of the values from cboState select element. (For example 2 is not a valid value)

But this is right in your example so the second point is not to have parameters in your post request which are not valid. Meaning in this example you shouldn't add cboCity when you post to __EVENTTARGET cboState.

Long story short you need to use this form fields:

FORM_FIELDS = {
    '__EVENTTARGET': 'cboState',
    '__EVENTARGUMENT': '',
    '__LASTFOCUS': '',
    '__VIEWSTATE': view_state,
    '__VIEWSTATEGENERATOR': view_state_generator,
    '__EVENTVALIDATION': event_validation,
    'cboState': '3',
    'hdDealerMaps': 'True',
}

The updated version of the script: https://gist.github.com/fliiiix/ea365b96f5ab4ec4d345