Multiple goals from one recipe and parallel execution

Question

I have a project that includes a code generator that generates several .c and .h files from one input file with just one call to the code generator. I have a rule that has .c and .h files as several targets, the input file as a prerequisite, and the recipe is a call to the code generator. Then I have additional rules for compiling and linking the generated .c files.

This works fine with -j factor 1, but if I increase the coefficient j, I find that I get multiple calls to the code generator, down to the -j factor or the number of expected target files, whichever is the smallest, This is bad, because that multiple calls to the code generator can cause failures due to the fact that the generated code is written several times.

I will not post my actual (large) code here, but I managed to create a small example that seems to exhibit the same behavior.

The Makefile is as follows:

  output.concat: output5 output4 output3 output2 output1
     cat $ ^> $@

 output1 output2 output3 output4 output5: input
     ./frob input

 clean:
     rm -rf output *

Instead of the code generator for this example, I wrote a simple shell script, frob , that generates multiple output files from a single input file:

  #! / bin / bash

 for i in {1..5};  do
     {
     echo "This is output $ {i}, generated from $ {1}. input was:"
     cat $ {1}
     }> output $ {i}
 done

When I run this Makefile with factors other than one, I get the following output:

  $ make -j2 
 ./frob input
 ./frob input
 cat output5 output4 output3 output2 output1> output.concat
 $

We see that ./frob is called twice here, which is bad. Is there a way I can build this rule so that the recipe is called only once, even with the non-unity coefficient -j?

I examined a rule change, so that only one of the expected output files is the target, and then adds another rule without a prescription, so that its goal is the remaining expected output files, and the precondition is the first expected output file. But I’m not sure if this will work, because I don’t know if I can guarantee the order in which the files are created, and therefore can end up with circular dependencies.

Answer 1

Here's how make works to work. A rule like this:

 foo bar baz : boz ; $(BUILDIT) 

exactly equivalent to write these three rules:

 foo : boz ; $(BUILDIT) bar : boz ; $(BUILDIT) baz : boz ; $(BUILDIT) 

There is no way (in GNU make) to define an explicit rule with the required characteristics; this is that one recipe call will build all three goals.

However, if your output files and your input file have a common base, you can write a template rule as follows:

 %.foo %.bar %.baz : %.boz ; $(BUILDIT) 

It is strange that for implicit rules with multiple goals, GNU make assumes that a single recipe call will build all goals and behave exactly the way you want.

Answer 2

@MadScientist's answer is promising - I think I could use this. Meanwhile, I played with this and came up with another possible solution, as I hinted at a question. I can split the rule into two as follows:

 INPUT_FILE = input OUTPUT_FILES = output5 output4 output3 output2 output1 OUTPUT_FILE1 = $(firstword $(OUTPUT_FILES)) OUTPUT_FILES_REST = $(wordlist 2,$(words $(OUTPUT_FILES)),$(OUTPUT_FILES)) $(OUTPUT_FILE1): $(INPUT_FILE) ./frob $< touch $(OUTPUT_FILES_REST) $(OUTPUT_FILES_REST): $(OUTPUT_FILE1) 

Providing only one output file as a target resolves a possible parallelism problem. Then we make this output file mandatory for the rest of the output files. It is important to note that in the frob recipe we touch all output files except the first, so we guarantee that the first will have an older timestamp than all the others.

Answer 3

Correctly generate and update several ab targets ab in parallel, make -j from the i1 i2 input files:

 all: abc .INTERMEDIATE: d a: d b: d c: d d: i1 i2 cat i1 i2 > a cat i1 i2 > b cat i1 i2 > c 

• If any of a,b,c missing, the pseudo-target d redone. The d file is never created; a single rule for d avoids multiple concurrent recipe calls.

• .INTERMEDIATE ensures that the missing d file does not invoke the recipe d .

• Some other ways for several purposes are in the book "John Graham-Cumming - GNU Make Book" p .92-96.