An elegant way to detect the end of an asynchronous recursive file search?

Question

Given a recursive subroutine in single threaded environment which starts numerous asynchronous I/O operations and registers callback functions for each of them. This callbacks will be called on the completion of the operations.

Which ways are recommended to detect when all of the I/O operations are finished, in other words all of the callback have been called.

Since the code runs on a single thread, the most primitive solution would be to initialize a counter variable to 0, and increment this counter before every I/O operations and decrement it at tail of the callback functions. One can use events or actions (like in elm), however that solution would be very similar.

One possible optimization: if multiple I/O operation chained together (So one and only one I/O operation is started in the other I/O operations callback function), than it would be suffice to increment the counter at the beginning of the chain and decrement at the end of the chain, However one should aware the case when multiple I/O operation is started in one callback function, then all branch should be handled as a different chain.

Are there any more elegant way to handle this?

Answer 1

Recursive file search is akin to walking a tree structure. You can go depth-first or breadth-first, you know you have traversed the entire tree when the stack/queue you are using for pending items is empty. BUT With async APIs, like that of nodejs, recursion is a bit tricky than simply calling the function again.

I'm going to give a rough idea:

1. Define a stack for directories that are to be searched, add starting/root directory to it.

2. Define a function that takes a directory path and returns a readable Stream. This stream emits each found child directory/file, complete with stats.

3. Take/pop a directory from #1 and convert it into stream using #2.

4. On stream from #3, subscribe to data events and if a directory, add/push it to #1. Also send the data to whatever is consuming the search results.

5. On stream from #3, subscribe to end event and goto #3.

6. When you goto #3 and find there is nothing more to pop from #1, the search is completed.

In essence, each pass waits for listing and stating files of 1 directory, before the next pass.