Easy development environment

Question

Recently, I was working on a reorganization of a system that processes client data packets. The system performs a series of steps, each of which uses the files of the previous steps (and sometimes the data in memory) and produces its own output in the form of files or data. Sometimes the output for a particular step is already available. I have to be careful to make sure that when one step fails, we continue to perform all possible steps (those that are independent of the failed step), so that the final conclusion is as complete as possible. In addition, not all steps must be performed in all situations.

Previously, relationships were implicit in the code structure. For instance:

void processClientData() { try { processA(); } catch(Exception e) { log.log(Level.SEVERE, "exception occured in step A", e); processC(); // C doesn't depend on A, so we can still run it. throw e; } processB(); processC(); //etc... for ~20 steps } 

I modified this to make explicit dependencies dependent, consistent error handling, etc., by introducing Tasks:

 public interface Task { List<Task> getDependencies(); void execute(); //only called after all dependencies have been executed } public class TaskRunner { public void run(Set<Task> targets) { // run the dependencies and targets ala ANT // make sure to run all possible tasks on the "road" to targets // ... } } 

It starts to look very much like a very irrigated version of a dependency management build system (ANT, being most familiar to me). I don't want to drag ANT for this kind of thing, and I certainly don't want to write out XML.

I have my system and it works (mostly), but it is still a bit hacked, and since then I have been thinking about how much I hate reinventing the wheel. I would expect this to be a fairly common problem that has been solved many times by people smarter than me. Alas, several hours of work on Google did not appear.

Is there a library that implements such things without being a really heavy prefab system? I also appreciate any pointers, including libraries in other languages ​​(or even new systems), from which I should draw inspiration.

EDIT: I appreciate the suggestions (and I will pay due attention to them), but I'm really NOT looking for an “assembly system” as such. What I'm looking for is more like the kernel of the build system, which I could just call directly from Java and use as a small library with low loads to perform the specified dependency analysis, task execution, and resource management. As I said, I have existing (working) code in pure Java, and I don’t want to enter the XML and all the baggage that comes with it, without a compelling reason . . p>

Answer 1

Take a look at the jsr166 fork / join framework. It seems to me that this is exactly what you are trying to accomplish.

http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ForkJoinTask.html

This is included in JDK7, but is available as a separate can for 5 and 6. If I were not on my tablet, I would write a more complete example. Maybe someone else can expand in the meantime.

 public class DependencyTreeTask extends RecursiveAction { private final List<DependencyTreeTask> dependencies = new ArrayList<Task>(); public void addDependency(DependencyTreeTask t) { dependencies.add(t) } public void compute() { invokeAll(dependencies); } } ... // build tree... DependencyTreeTask root = ... ForkJoinPool pool = new ForkJoinPool(); pool.invoke(root); 

You also need to make sure that your schedule is not connected, but there is a well-known set of algorithms for determining it.

Answer 2

At its core, the build system does 3 things. He manages the addiction, checks whether something is “built” or not, and “builds” things that are not built.

Dependency management is not just simple topological sorting. The rest iterates through the tasks in a dependent order and processes them.

You can easily create something like:

 BuildSystem bs = new BuildSystem(); bs.addTask(new Task1()); bs.addTask(new Task...); bs.addTask(new TaskN()); bs.build(); public void build() { List<Task> sortedTasks = topologicalTaskSort(tasks); for(Task t : sortedTasks) { if (t.needsBuilding()) { t.execute(); } } } 

If you do not need to externalize the task list, then there is no reason for an XML file or anything else.

Topological sorting allows you to simply add tasks to the list and let the system parse things. Not a problem with 4 tasks, a big problem with dozens of tasks.

A variety fails if it detects a dependency cycle, so when you get this control.

Something like this is "too simple" to need structure. I don’t know how you manage your dependencies right now.

Answer 3

I would think of writing a Maven plugin, it’s not so difficult and much lighter weight , because you only need to provide the appropriate special logic. All infrastructure is provided by Maven. Maven 3 will even give you things like parallel builds, where your plugin supports it for free, among all the other things it provides.

One of the main goals of Maven 3 was rewriting to make it as easy as possible to integrate the workflow engine into your own projects.

Answer 4

I have heard that Jenkins is used for these kinds of things in addition to this primary role of “assembly”. I just started using Jenkins, so I can’t say for sure if he will do what you need. I am still impressed with this. It is relatively easy to use and has many configuration options. For him there are a large number of plugins. Just launch Jenkins and go to the plugins page to view the list and install them.

Answer 5

Your code reminds me of iwant , the java build engine I developed. You can declare your target definitions using dependencies using free Java, and in addition to the usual use from the command line (or ant script), you can also embed your assembly in a Java program.