Return predicate from nested async function call

Question

I need to filter an array, depending whether or not each element's sub-element, obtained by retrieving data from an external API, matches a criteria.

The code might be clearer than an explanation :

//Suppose there is an array filled
//_.filter(array, predicate)
var filtered_array = _.filter(array,function(element){
   //Async operation
   webservices.doStuff(element, function(sub_element){
        //This return is meant for the filter, but returns on .doStuff callback
       return sub_element.thing === "ok";
   });
});

The problem here, is I don't know how to return the predicate's result here, as webservices.doStuff is asynchronous!

Any idea?

Answer 1

I would use Q or async to do this. Consider following example, you wrap each web service call in a promise and define another promise that will be resolved when all promises are resolved. Then you filter results ( ie array of all returns from all calls) according to your logic.

 var source = [{a : 1}, {a:2}...] // your data

 var all = Q.all(source.map((o) => Q.Promise((resolve, reject,notify) 
                   => webService.doStuff(o, resolve, reject))))

 all.then((ret) => _.filter(ret.data, (elem) => elem.thing === "ok"))
    .then((filtered) => ... // do something with filtered )

Most likely your web service proxy already have binding with Q or another promise library. So your code is going to be simpler, something like:

var all = Q.all(source.map(webService.doStuff))
           .then(filter).then(process)

Or at least you can refactor it to be that simple.

Overall due to non-blocking-accept-callback-and-go nature of node.js environment, it is essential to master one promise library or another, async and Q are two most popular to the best of my knowledge.

(Update: right now it seems to be bluebird, and it is fastest too.)

Having either of them allow you to come up with much easier design when dealing with async calls.

Answer 2

You should take a look in the async.js library https://github.com/caolan/async which can do alot of async requests, and issue a single callback when all the request are done, and provide you with a list of all the results, which you then can apply you filter function on. I cannot remember you you can provide a predicate function right away describe which results that should be in the final resultset, but it should be do able to do such a thing, without too much work, if the library doesn't already support it.

Take a look at the filter function in the async.js documentation.