Gevent incremental process through non blocking joinall()

Question

Here I want to make some modificatins for my setting.

I want response from multiple API calls within a single request made to my server. from all these API calls I want to combine results and return them as a response. Until here pretty much everything follows as given in examples of gevent documentation and over here. Now the catch here is that I want to pass response in incremental way, so if first API call has returned the result I will return this result to frontend in one long waited request and then wait for other API calls and pass them in same request to frontend.

I have tried to do this through code but I dont know how to proceed with this setting. The gevent .joinall() and .join() block untill all the greenlets are finished getting responses.

Any way I can procceed with gevent in this setting ?

Code I am using here is given on link https://bitbucket.org/denis/gevent/src/tip/examples/concurrent_download.py . Here the .joinall() in the last statement waits until all urls have complete giving responses, I want it to be non blocking so that I can process the responses in the callback function print_head() and return them incrementally.

#!/usr/bin/python
# Copyright (c) 2009 Denis Bilenko. See LICENSE for details.

"""Spawn multiple workers and wait for them to complete"""

urls = ['http://www.google.com', 'http://www.yandex.ru', 'http://www.python.org']

import gevent
from gevent import monkey

# patches stdlib (including socket and ssl modules) to cooperate with other greenlets
monkey.patch_all()

import urllib2


def print_head(url):
    print ('Starting %s' % url)
    data = urllib2.urlopen(url).read()
    print ('%s: %s bytes: %r' % (url, len(data), data[:50]))

jobs = [gevent.spawn(print_head, url) for url in urls]

gevent.joinall(jobs)

Answer 1

If you want to collect results from multiple greenlets, then modify print_head() to return the result and then use .get() method to collect them all.

Put this after joinall():

total_result = [x.get() for x in jobs]

Actually, joinall() is not even necessary in this case.

If print_head() looks like this:

def print_head(url):
    print ('Starting %s' % url)
    return urllib2.urlopen(url).read()

Then total_result will be a list of size 3 containing the responses from all the requests.