Python urllib2.urlopen bug: timeout error brings down my Internet connection?

Question

I don't know if I'm doing something wrong, but I'm 100% sure it's the python script brings down my Internet connection.

I wrote a python script to scrape thousands of files header info, mainly for Content-Length to get the exact size of each file, using HEAD request.

Sample code:

class HeadRequest(urllib2.Request):
    def get_method(self):
        return "HEAD"

response = urllib2.urlopen(HeadRequest("http://www.google.com"))
print response.info()

The thing is after several hours running, the script starts to throw out urlopen error timed out, and my Internet connection is down from then on. And my Internet connection will always be back on immediately after I close that script. At the beginning I thought it might be the connection not stable, but after several times running, it turned out to be the scripts fault.

I don't know why, this should be considered as a bug, right? Or my ISP banned me for doing such things? (I already set the program to wait 10s each request)

BTW, I'm using VPN network, does it have something to do with this?

Answer 1

I'd guess that either your ISP or VPN provider is limiting you because of high-volume suspicious traffic, or your router or VPN tunnel is getting clogged up with half-open connections. Consumer internet is REALLY not intended for spider-type activities.

Answer 2

"the script starts to throw out urlopen error timed out"

We can't even begin to guess.

You need to gather data on your computer and include that data in your question.

Get another computer. Run your script. Is the other computer's internet access blocked also? Or does it still work?

• If both computers are blocked, it's not your software, it's your provider. Update Your Question with this information, and how you got it.

• If only the computer running the script is stopped, it's not your provider, it's your OS resources being exhausted. This is harder to diagnose because it could be memory, sockets or file descriptors. Usually its sockets.

You need to find some ifconfig/ipconfig diagnostic software for your operating system. You need to update your question to state exactly what operating system you're using. You need to use this diagnostic software to see how many open sockets are cluttering up your system.