Q - running a series of promises and defining dependencies between them in a DAG

Question

I would like to process a series of data where each of them can be used as inputs to the others.

For instance:

var batch = [ {"id":"a1","depends":[],"data":{"some":"data a1"}}, {"id":"b1","depends":["a1"],"data":{"some":"data b1"}}, {"id":"b2","depends":["a1"],"data":{"some":"data b2"}}, {"id":"c1","depends":["b1","b2"],"data":{"some":"data c1"}}, {"id":"x1","depends":[],"data":{"some":"data x1"}}, ]; 

This means that as soon as a1 is completed, its output will be sent to both b1 and b2 ; and when they are completed, both of their outputs will be sent to c1 (only after they are completed. x1 can be executed in parallel with all a1 , b1 , b2 and c1 ; and b1 can be executed in parallel with b2 , because it does not define between them.

Upon completion of c1 and x1 , and, consequently, the completion of all 5 of them, the result of all five should be returned.

We will assume that no circular dependencies are defined and, therefore, is a directed acyclic graph (DAG)

I would like to know how to implement this using Q , because:

• All data processing will be asynchronous, and so I will need to use either callbacks or deferred, and promises; and I prefer the latter
• Promises can double as a convenient way to define edges in a graph

However, I could not go through this conceptual stage.

 var doPromises = {}; var doData = function(data, dependsResultsHash, callback) { //Not real processing, simply echoes input after a delay for async simulation purposes var out = { echo: { data: data, dependsResultsHash: dependsResultsHash } }; setTimeout(function() { callback(out); }, 1000); }; var doLine = function(id, depIds, data) { var deferred = Q.defer; var dependsPromises = []; for (var i = 0; i < depIds.length; ++i) { var depId = depIds[i]; dependPromise = doPromises[depId]; dependsPromises.push(dependPromise); } Q.all(dependsPromises).then(function(dependsResults) { var dependsResultsHash = {}; for (var i = 0; i < depIds.length; ++i) { var depId = depIds[i]; var depResult = dependsResults[i]; dependsResultsHash[depId] = depResult; } doData(data, dependsResultsHash, function(result) { deferred.resolve(result); }); }); return deferred.promise; } var doBatch = function(batch) { var linePromises = []; for (var i = 0; i < batch.length; ++i) { var line = batch[i]; var linePromise = doLine(line.id, line.depends, line.data); linePromises.push(linePromise); doPromises[line.id] = linePromise; } Q.all(linePromises).then(function(lineResults) { console.log(lineResults); deferred.resolve(lineResults); }); }; doBatch(batch); 

(Note that this code has not been verified, and I do not expect it to work, just to illustrate the points needed for my question.)

I'd like to know:

• Am I doing it right? I completely lost the point with the Q library. or with deferrals and promises?

• My main problem is the doData function:

   -- Is the way that I have selected the promises of the lines depended upon from the global list of promises `doPromises` ok? -- Is the way that I have obtained the results of the lines depended upon, and inpterpreted that OK? 

• Using the doBatch function:

   -- I have a local array for `linePromises` and an external hash for `doPromises`, and I feel that these should be combined. How can I do this correctly? 

• General

   -- The code above presently assumes that all `deferred`s will eventually keep their `promise`s. What if they fail or throw an exception; how do I make my code more robust in handling this? -- I have used a closure allow acces to `doPromises` in both `doBatch` and `doLine`, and it seems a little odd here, is there a better way to do this? 

Answer 1

I created a library that does this:

qryq is a NodeJs library that allows you to express a series of queries and determine the relationships between them, either in parallel, sequentially, or in a directed acyclic graph.

Answer 2

I recently created a module called dagmise , which I plan to use to create a build system that uses promises as jobs. I finished creating graph function nodes returning promises. When you visit a node, the function in it is evaluated, and the returned promise replaces the value of the node. Therefore, even if node is visited several times, the function is executed only once.

I started with the idea that promises should be edges, but now I think they are simpler in nodes. Otherwise, you really have two types of objects on your graphs (nodes / states and edges / promises), which complicates the situation a bit.

Answer 3

I just implemented my own library for this: promise-dag .

I was not satisfied with the indicated alternatives ( dagmise , qryq ) for the following reasons:

• They impose a concrete implementation of promises. promise-dag compatible with any implementation of Promise without any code dependency.
• They provide a free API (chaining methods). I prefer the data-oriented API, which is more programmable and transparent, which simplifies the integration of calculation graphs using the usual data structure functions ( _.extend() , _.pick() , ...), as well as adding custom tools ( profiling / tracing / logging / etc.).