Python clicking a button on webpage (back end)

Question

I am trying using Python to simulate the login to company email.

The scripts works find and result page shows an already login sign.

import urllib
import urllib2
import mechanize
import cookielib

br = mechanize.Browser()

cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

br.set_handle_equiv(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

br.addheaders = [('User-agent', 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)')]

url = 'http://www.company.com/'
form_data = {'name_file': 'THENAME', 'password_field': 'THEPASSWORD'}
params = urllib.urlencode(form_data)
response = br.open(url, params)

However I need to click the “GoTo Email” button on the webpage to enter the email dashboard. Note here the web address doesn’t change, and not redirecting to any other page, before and after clicking the button.

The HTML script is showed as below. I think it’s a button.

id=mail1_btnGoto class=btn onmouseover="this.className='btnOver'" onmouseout="this.className='btn'" name=mail1$btnGoto value="GoTo Email" type=submit>

I thought to use winapi to simulate a mouse click but it’s silly because it controls mouse at the front end only. Selenium isn’t a solution in this case because I want to have the script run at the back end.

How can I have the button ‘click’ on the webpage?

Answer 1

It seems the email dashboard is driven by Javascript, so you cannot simply use winapi to simulate mouseclick without evaluating script.

Generally there are two workarounds:

1. Use full-feature browser driver. As you mentioned above, selenium is a good choice across many programming language. The webdriver does not need opening browsers manually and can be fully controlled by scripts. You can try ghost driver instead. It uses PhantomJS and should run in backend server.(But installing phantomjs is required)

2. mock the request. Because logining will usually invoke a http/https request. You can use python to mock that request. You can use http debbuging tools like fiddler, wireshark or Chrome web inspector to capture the information the browser sent to authentication server.

I tries to be specific and detailed. But due to the diversity of web crawling a step by step guide is beyond my reach.