How to write a python script to keep trying a webpage until it opens

Question

We are waiting for my sister's result. And as it happens, a lot, with the Indian govt. the server is slow, traffic is heavy.

So, I thought of writing a python program, to keep trying until the server responds to the Http request. But the program:

import urllib
i=1
f = open("C:/Users/DELL/Desktop/neetpg.html",'w')
while(True):
    try:
        page = urllib.urlopen("http://www.nbe.gov.in/asr/neet_pdf/")
        print "Done"
        break
    except:
        print i
        i += 1
        continue
f.write(page.read())

print "check"

But the program, doesn't run properly. I tried replacing the url with facebook.com, it still prints out numbers.

Moreover what I'd like, is to achieve that if the server does respond, the webpage loads the js and css files along with the html file and all this should open in a browser.

I also took a hint from http://docs.python.org/2/library/webbrowser.html and changed the program to:

import webbrowser
i=1
while(True):
    try:
        webbrowser.open("http://www.nbe.gov.in/asr/neet_pdf/")
        print "Done"
        break
    except:
        print i
        i += 1
        continue
print "check"

But all this does, is opens a new window in my default web browser, and sets the url to what is given, and "opens it". Meanwhile, printing Done and Check on the python shell.

The web browser having not received a response from the server, displays could not connect to www.nbe.gov.in.

How to achieve this ?

EDIT: Just saw that the facebook.com script worked after all. It took it approx 15 tries, and then it happened. The .html file is written properly. With all the CSS and probably the JS too.

Why is it so, that it took so many tries, while, I can simply open facebook.com from the browser easily.

Answer 1

Give selenium a try.

The idea is to keep opening the page until the driver sees the right title. And if it's there, just break the loop and leave the page opened:

from selenium import webdriver


driver = webdriver.Firefox()
while True:
    driver.get("http://www.nbe.gov.in/asr/neet_pdf/")

    if 'NEET-PG' in driver.title:
        break

Hope that helps.