Pmap and number of threads

Question

user=> (.. Runtime getRuntime availableProcessors) 2 

And evaluating this example: http://clojuredocs.org/clojure_core/clojure.core/pmap#example_684 I get

 user=> (time (doall (map long-running-job (range 4)))) "Elapsed time: 12000.621 msecs" (10 11 12 13) user=> (time (doall (pmap long-running-job (range 5)))) "Elapsed time: 3000.454 msecs" (10 11 12 13 14) user=> (time (doall (pmap long-running-job (range 32)))) "Elapsed time: 3014.969 msecs" (10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 3839 40 41) user=> (time (doall (pmap long-running-job (range 33)))) "Elapsed time: 6001.526 msecs" (10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42) 

I wonder why I have to go through 33 to wait 33 seconds. for the result. pmap create 2 (available processors) + 2 threads, yes? I believe that when passing (range 5) it will be completed in 6 seconds. Why is it different?

Answer 1

In fact, pmap not subject to the "processors + 2" limit. This is the result of how regular map and future macros work:

• future uses a pool of cached streams, which has no size limit;

• map creates a sequence with alternation, that is, one that always forces 32 elements at a time, even if only the heap at the beginning of the fragment is actually consumed by the caller.

The end result is that futures in pmap run in parallel in blocks 32.

Please note that this does not violate the contract specified in pmap docstring. On the other hand, the code can lead to the assumption that the limit "processors + 2" will be respected - as if map were written naively. In fact, pmap may well precede the transition to chunked seqs, although I'm not quite sure it has been a while.