python mechanize code to get webpage screenshot - but form submit NOT WORK

Question

I browsed to this site: http://ctrlq.org/screenshots/ and found out that the form on this page can give the webpage screenshot of any URL we give it.

Now, I want to to try the same using mechanize to see if I can achieve it through submitting the form through python code.

If the form submit is successful, I should get

Congratulations, the screenshot image of http://www.gnu.org is ready. Download Image

But, my submit is unsuccessful to get the expected result. Where am I going wrong?

Here is the code:

import mechanize
import cookielib
from BeautifulSoup import BeautifulSoup
import re


# Browser
br = mechanize.Browser()

# Cookie Jar
cj = cookielib.LWPCookieJar()
br.set_cookiejar(cj)

# Browser options
br.set_handle_equiv(True)
#br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

# Follows refresh 0 but not hangs on refresh > 0
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)

# User-Agent (this is cheating, ok?)
br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')]

# The site we will navigate into, handling it's session
br.open('http://ctrlq.org/screenshots/')

html = br.response().read()
soup = BeautifulSoup(html)

hidden_str = str(soup.find('input', { "name" : "labnol" }))
m = re.search('value="(.*?)"', hidden_str)
hidden_val = m.group(1)


print hidden_val


br.select_form(nr=0)

br['url'] = "http%3A%2F%2Fwww.gnu.org"

for form in br.forms():
    print '---------------'
    print form

br.submit()

html2 = br.response().read()


##print html2

print 'if congrat.. exists in result'
print 'congrat' in html2.lower()

And, here is the result:

>>> ================================ RESTART ================================
>>> 
2f043008c3ecfa0a86ea8a9ed8a19916
---------------
<POST http://ctrlq.org/screenshots/ application/x-www-form-urlencoded
  <TextControl(url=http%3A%2F%2Fwww.gnu.org)>
  <SubmitButtonControl(<None>=) (readonly)>
  <HiddenControl(labnol=2f043008c3ecfa0a86ea8a9ed8a19916) (readonly)>>

False
>>> 

On a side note, what is the best way to get a screenshot of a webpage with given url using python. I don't want to use pyqt as I have no idea how that works. Any other simpler alternatives with built-in modules or simpler modules.

Answer 1

Long time from asking the question. But if someone needs can use this:

from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://www.python.org/')
driver.save_screenshot('screenshot.png')
driver.quit()

You have to download the selenium package from here: https://pypi.python.org/pypi/selenium Or run command

pip install selenium

Note: You need also geckodriver.exe. It should be found in the same location as your script. Download link for gecko: https://github.com/mozilla/geckodriver/releases .

Other usefull documentation for selenium here: http://selenium-python.readthedocs.io/faq.html Selenium has many other features that mechanize doesn't: Selenium Webdriver vs Mechanize

Answer 2

You can try this on your linux machine to have automated screenshots on your own.