Checking if website responds in python using a browser user agent

Question

I am trying to come up with a script to check if a domain name resolves to its IP address via dns; using a python script I wrote.

I want to be able to do this in a few sequential loops, however after trying to run a loop once, the second time i run the script, the names that previously returned a successful dns resolution response, now do not.

Below is my script:

#! C:\Python27
import socket,time

localtime = time.asctime( time.localtime(time.time()) )


def hostres(hostname):
    print "Attempting to resolve " + hostname 
    try:
        socket.gethostbyname(hostname)
        print "Resolved Successfully!"
    except socket.error:
        print "Could Not Resolve"

print "*************************************************"
print "Website loop starting.."
print "Local current time :", localtime
print "*************************************************"
print ""

text_file = open("sites.txt", "r")
lines = text_file.readlines()
for line in lines:
    hostres(line)
text_file.close()

The contents of the text file are:

www.google.com
en.wikipedia.org
www.youtube.com
us.gamespot.com

I am thinking it is to do with these domains servers recognizing the script as a "bot" rather than a legitimate end-user, would it be correct to assume this?

If so, how can I still check if the dns name resolves by looking up the name of the website (or IP, does not matter) and be able to run this without getting a false reading of "request failed" despite the fact that the service is fully accessible from a browser?

Answer 1

Several problems in this question.

1. You are not checking if "a website responds" you are testing DNS resolution. All your DNS requests go to a single name server, your LDNS resolver. If all of them resolve, it still says nothing about the status of the website. Also, since you aren't actually talking to these website, they have no way of knowing you're a bot. They can only detect this (based on the HTTP user-agent header) if you make a HTTP request.
2. Regarding your code problem, you need to trim the newline character before you can do a socket.gethostbyname() on it. Replace socket.gethostbyname(hostname) with socket.gethostbyname(hostname.rstrip()) and you'll be fine.