sequential calls of methods asynchronously

Question

I have a list of methods I am calling in a method, as follows:

this.doOneThing();
someOtherObject.doASecondThing();
this.doSomethingElse();

When this is synchronous, they are executed one after the other, which is required. But now I have someOtherObject.doASecondThing() as asynchronous, and I might turn doOneThing as async too. I could use a callback and call that.doSomethingElse from inside the callback:

var that = this;
this.doOneThing( function () { 
                    someOtherObject.doASecondThing(function () {
                        that.doSomethingElse();
                    });
                  });

However, because the sequence is growing, it seems a little messy to have callbacks call each other, for some reason it makes the sequence not look as obvious as before, and the indentation might grow with the number of methods called in the sequence.

Is there a way to make this look better? I could use the observer pattern also but it doesn't make things very obvious either, in my opinion.

Thanks,

Answer 1

Continuations, and why they're causing callback spaghetti

Writing in callbacks forces you to write in sometime akin to "continuation-passing style" (CPS), an extremely powerful but difficult technique. It represents a total inversion of control, literally turning a computation "upside-down". CPS makes your code's structure explicitly reflect the control flow of your program (sometimes a good thing, sometimes a bad thing). In effect, you are explicitly writing down the stack out of anonymous functions.

As a prerequisite for understanding this answer, you may find this useful:

http://matt.might.net/articles/by-example-continuation-passing-style/

For example this is what you are doing:

function thrice(x, ret) {
    ret(x*3)
}
function twice(y, ret) {
    ret(y*2)
}
function plus(x,y, ret) {
    ret(x+y)
}

function threeXPlusTwoY(x,y, ret) {
    // STEP#1
    thrice(x,                 // Take the result of thrice(x)...
        function(r1) {        // ...and call that r1.
            // STEP#2
            twice(y,            // Take the result of twice(y)...
                function(r2) {  // ...and call that r2.
                    // STEP#3
                    plus(r1,r2,   // Take r1+r2...
                        ret       // ...then do what we were going to do.
                    )
                }
            )
        }
    )
}

threeXPlusTwoY(5,1, alert);  //17

As you've complained about, this makes for fairly indented code, because closures are the natural way to capture this stack.

---

Monads to the rescue

One way to unindent CPS is to write "monadically" like in Haskell. How would we do that? One nice way of implementing monads in javascript is with the dot-chaining notation, similar to jQuery. (See http://importantshock.wordpress.com/2009/01/18/jquery-is-a-monad/ for an amusing diversion.) Or we can use reflection.

But first we need a way to "write down the plumbing", and THEN we can find a way abstract it away. Tragically it's sort of hard to write a generic monad syntax in javascript, so I will use lists to represent computations.

 

// switching this up a bit:
// it's now 3x+2x so we have a diamond-shaped dependency graph

// OUR NEW CODE
var _x = 0;
var steps = [
    [0,  function(ret){ret(5)},[]],  //step0:
    [1,  thrice,[_x]],               //step1: thrice(x)
    [2,  twice,[_x]],                //step2: twice(x)
    [3,  plus,[1, 2]]                //step3: steps[1]+steps[2] *
]
threeXPlusTwoX = generateComputation(steps);

//*this may be left ambiguous, but in this case we will choose steps1 then step2
// via the order in the array

That's kinda ugly. But we can make this UNINDENTED "code" work. We can worry about making it prettier later (in the last section). Here our purpose was to write down all the "necessary information". We'd like an easy way to write each "line", along with a context we can write them in.

Now we implement a generateComputation which generates some nested anonymous functions which would perform the above steps in-order if we executed it. This is what such an implementation would look like:

function generateComputation(steps) {
    /*
    * Convert {{steps}} object into a function(ret), 
    * which when called will perform the steps in order.
    * This function will call ret(_) on the results of the last step.
    */
    function computation(ret) {
        var stepResults = [];

        var nestedFunctions = steps.reduceRight(
            function(laterFuture, step) {
                var i            = step[0];  // e.g. step #3
                var stepFunction = step[1];  // e.g. func: plus
                var stepArgs     = step[2];  // e.g. args: 1,2

                console.log(i, laterFuture);
                return function(returned) {
                    if (i>0)
                        stepResults.push(returned);
                    var evalledStepArgs = stepArgs.map(function(s){return stepResults[s]});
                    console.log({i:i, returned:returned, stepResults:stepResults, evalledStepArgs:evalledStepArgs, stepFunction:stepFunction});
                    stepFunction.apply(this, evalledStepArgs.concat(laterFuture));
                }
            },
            ret
        );

        nestedFunctions();
    }
    return computation;
}

Demonstration:

threeXPlusTwoX = generateComputation(steps)(alert);  // alerts 25

sidenote: reduceRight semantics implies the steps on the right will be more deeply nested in functions (further in the future). FYI for those not familiar, [1,2,3].reduce(f(_,_), x) --> f(f(f(0,1), 2), 3), and reduceRight (due to poor design considerations) is actually equivalent to [1.2.3].reversed().reduce(...)

Above, generateComputation made a bunch of nested functions, wrapping them in one another in preparation, and when evaluated with ...(alert), unpeeled them one-by-one to feed into the computation.

sidenote: We have to use a hack because in the previous example, we used closures and variable names to implement CPS. Javascript does not allow sufficient reflection to do this without resorting to crafting a string and evaling it (ick), so we eschew functional style temporarily and opt for mutating an object that keeps track of all parameters. Thus the above more closely replicates the following:

var x = 5;
function _x(ret) {
    ret(x);
}

function thrice(x, ret) {
    ret(x*3)
}
function twice(y, ret) {
    ret(y*2)
}
function plus(x,y, ret) {
    ret(x+y)
}

function threeXPlusTwoY(x,y, ret) {
    results = []
    _x(
        return function(x) {
            results[0] = x;

            thrice(x,                 // Take the result of thrice(x)...
                function(r1) {        // ...and call that r1.
                    results[1] = r1;

                    twice(y,            // Take the result of twice(y)...
                        function(r2) {  // ...and call that r2.
                            results[2] = r2;

                            plus(results[1],results[2],   // Take r1+r2...
                                ret       // ...then do what we were going to do.
                            )
                        }
                    )
                }
            )

        }
    )
}

---

Ideal syntax

But we still want to write functions in a sane way. How would we ideally like to write our code to take advantage of CPS, but while retaining our sanity? There are numerous takes in the literature (for example, Scala's shift and reset operators are just one of the many ways to do so), but for sanity's sake, let's just find a way to make syntactic sugar for regular CPS. There are some possible ways to do it:

 

// "bad"
var _x = 0;
var steps = [
    [0,  function(ret){ret(5)},[]],  //step0:
    [1,  thrice,[_x]],               //step1: thrice(x)
    [2,  twice,[_x]],                //step2: twice(x)
    [3,  plus,[1, 2]]                //step3: steps[1]+steps[2] *
]
threeXPlusTwoX = generateComputation(steps);

...becomes...

• If the callbacks are in a chain, we can easily feed one into the next without worrying about naming. These functions only have one argument: the callback argument. (If they didn't, you could curry the function as follows on the last line.) Here we can use jQuery-style dot-chains.

 

// SYNTAX WITH A SIMPLE CHAIN
// ((2*X) + 2)
twiceXPlusTwo = callbackChain()
    .then(prompt)
    .then(twice)
    .then(function(returned){return plus(returned,2)});  //curried

twiceXPlusTwo(alert);

• If the callbacks form a dependency tree, we can also get away with jQuery-style dot-chains, but this would defeat the purpose of creating a monadic syntax for CPS, which is to flatten the nested functions. Thus we won't go into detail here.

• If the callbacks form a dependency acyclic graph (for example, 2*x+3*x, where x is used twice) we would need a way to name the intermediate results of some callbacks. This is where it gets interesting. Our goal is to try to mimic the syntax at http://en.wikibooks.org/wiki/Haskell/Continuation_passing_style with its do-notation which "unwraps" and "rewraps" functions into and out of CPS. Unfortunately the [1, thrice,[_x]] syntax was the closest we could easily get to that (and not even close). You could code in another language and compile to javascript, or using eval (queue the ominous music). A bit overkill. Alternatives would have to use strings, such as:

 

// SUPER-NICE SYNTAX
// (3X + 2X)
thriceXPlusTwiceX = CPS({
    leftPart: thrice('x'),
    rightPart: twice('x'),
    result: plus('leftPart', 'rightPart')
})

You can do this with only a few tweaks to the generateComputation I described. First you adapt it to use logical names ('leftPart', etc.) rather than numbers. Then you make your functions actually lazy objects which behave like:

thrice(x).toListForm() == [<real thrice function>, ['x']]
or
thrice(x).toCPS()(5, alert)  // alerts 15
or
thrice.toNonCPS()(5) == 15

(You would do this in an automated way with some kind of decorator, not manually.)

sidenote: All your callback functions should follow the same protocol about where the callback parameter is. For example if your functions begin with myFunction(callback, arg0, arg1, ...) or myFunction(arg0, arg1, ..., callback) they might not be trivially compatible, though if they aren't presumably you could do a javascript reflection hack to look at the source code of the function and regex it out, and thus not have to worry about it.

Why go through all that trouble? This allows you to mix in setTimeouts and prompts and ajax requests without suffering from "indent hell". You also get a whole bunch of other benefits (like being able to write a 10-line nondeterministic-search sudoku solver, and implementing arbitrary control-flow operators) which I will not go into here.