grequests
, like requests
, support an hook
keyword argument where you can assign a function that does something with the response object:
def do_something(response):
print response.status_code
unsentrequests=(grequests.get(u, hooks = {'response' : do_something}) for u in self.urls)
responses=grequests.map(unsentrequests)
I prefer to use requests
directly in a loop using gevent
to have a more explicit code:
import gevent.monkey
gevent.monkey.patch_socket()
from gevent.pool import Pool
import requests
def check_urls(urls):
def fetch(url):
response = requests.request('GET', url, timeout=5.0)
print "Status: [%s] URL: %s" % (response.status_code, url)
pool = Pool(20)
for url in urls:
pool.spawn(fetch, url)
pool.join()