Best template for handling an asynchronous loop in Node.js

Question

I am new to Node and try to use smart projects for a JSON-driven web application.

I have a bunch of data stored in Redis and I get it through node, unloading the results as they come from Redis. Here is a good example of what I'm doing:

app.get("/facility", function(req, res) { rc.keys("FACILITY*", function(err, replies) { res.write("["); replies.forEach(function (reply, i) { rc.get(reply, function(err, reply) { res.write(reply); if (i == replies.length-1) { res.write("]"); res.end(); } else res.write(","); }); }); }); }); 

Essentially, I get a set of keys from Redis, and then I request each, uploading the result to a semi-automatically generated JSON (the lines coming out of Redis are already in JSON). Now this works well, but I can't help but think that I == replies.length-1 is a bit untidy?

I could do all this with mget in Redis, but this is actually not quite what I am trying to get; this is how best to handle the asynchronous operation loop with forEach, streaming playback and graceful closing of the connection with res.end with the end of the loop.

Is this a better way, or is there a more elegant pattern that I could follow?

Answer 1

The above code may not do what you expect. You run each .get() in sequence, but they cannot call back sequentially, so the results can come out in any order. If you want to pass results, rather than collect them in memory, you need .get() in sequence.

I think the caolans asynchronous library makes this a lot easier. Heres one way you could use it to get each element in a sequence (warning, untested):

 app.get("/facility", function(req, res) { rc.keys("FACILITY*", function(err, replies) { var i = 0; res.write("["); async.forEachSeries(replies, function(reply, callback){ rc.get(reply, function(err, reply) { if (err){ callback(err); return; } res.write(reply); if (i < replies.length) { res.write(","); } i++; callback(); }); }, function(err){ if (err) { // Handle an error } else { res.end(']'); } }); }); }); 

If you don't care about ordering, just use async.forEach() .

If you don't mind collecting results and want them to come back sequentially, you could use async.map() like this (warning, also not verified):

 app.get("/facility", function(req, res) { rc.keys("FACILITY*", function(err, replies) { async.map(replies, rc.get.bind(rc), function(err, replies){ if (err) { // Handle an error } else { res.end('[' + replies.join(',') + ']'); } }); }); }); 

Answer 2

You can use the async library, it provides some convenient methods for looping, such as forEach:

forEach (arr, iterator, callback)

Applies an iterator function to each element of the array in parallel. An iterator is called with an element from the list and a callback for when it finishes. If the iterator passes an error callback, the main callback for the forEach function is immediately called with an error.

Note that since this function applies an iterator to each element in parallel there is no guarantee that the iterator functions will be completed in order.

Example

 // assuming openFiles is an array of file names and saveFile is a function // to save the modified contents of that file: async.forEach(openFiles, saveFile, function(err){ // if any of the saves produced an error, err would equal that error }); 

Answer 3

but I can't help but think that I == replies.length-1 is a little untidy?

I have heard many say this. So I would do it manually:

 app.get("/facility", function(req, res, next) { rc.keys("FACILITY*", function(err, replies) { if (err) return next(err); var pending = replies.length; res.write("["); replies.forEach(function (reply) { rc.get(reply, function(err, reply) { res.write(reply); if (!--pending) { res.write("]"); return res.end(); } res.write(","); }); }); }); }); 

Obviously, doing it manually is not very nice, so people decide to use it in a library or some other function. But whether you like it or not, you are doing an asynchronous parallel loop. :)

You can use the async library described earlier to hide nasty internals.