urllib3 Connection Management

Question

I am using urllib3 to query an online API over HTTP thousands of times (specifically, over 40,000) one after another.
I've noticed that in the beginning, the HTTP GET's from the API are really fast (< 500ms).
As more and more HTTP requests are being made, the response time slows down to the point of being a few seconds.

My code looks like this:

http = urllib3.PoolManager()
req = http.request(u'GET', URL)
doc = xml.dom.minidom.parseString(req.data)
nodes = doc.getElementsByTagName(ELEMENT_NAME)
for node in nodes:
    #process the node, this happens very very fast.

Am I missing anything? Thanks!

Answer 1

Perhaps the API you're hitting is throttling you?