How do you forcibly evaluate function input before testing a function in Criterion ? I am trying to compare some functions, but would like to eliminate the time for evaluating thunk input. This code uses unboxed vectors for input, which cannot be deep for Int vectors. Example code snippet below:
-- V is Data.Vector.Unboxed shortv = V.fromList [1..10] :: V.Vector GHC.Int.Int16 intv = V.fromList [1..10] :: V.Vector GHC.Int.Int32 main :: IO () main = defaultMain [ bench "encode ShortV" $ whnf encodeInt16V shortv ,bench "encode IntV" $ whnf encodeInt32V intv ]
The criterion test includes time to create short and inline inputs when monitoring functions. Measure the criteria below - it measures about ~ 400 ns for each function, which appears to include assembly time for input:
benchmarking encode ShortV mean: 379.6917 ns, lb 378.0229 ns, ub 382.4529 ns, ci 0.950 std dev: 10.79084 ns, lb 7.360444 ns, ub 15.89614 ns, ci 0.950 benchmarking encode IntV mean: 392.2736 ns, lb 391.2816 ns, ub 393.4853 ns, ci 0.950 std dev: 5.565134 ns, lb 4.694539 ns, ub 6.689224 ns, ci 0.950
Now, if the main section of the base code is changed below (by removing the second scanner function):
main = defaultMain [ bench "encode ShortV" $ whnf encodeInt16V shortv ]
Writing short messages is apparently evaluated before the encodeInt16V function is checked. This is really the desired result for me, because this test measures the execution time of the function, excluding the time to create the input. Inference criteria below:
benchmarking encode ShortV mean: 148.8488 ns, lb 148.4714 ns, ub 149.6279 ns, ci 0.950 std dev: 2.658834 ns, lb 1.621119 ns, ub 5.184792 ns, ci 0.950
Similarly, if I only compare the "encode IntV" test, I get ~ 150 ns for that too.
I know from the Criterion documentation that he is trying to avoid a lazy rating for more accurate benchmarking. It makes sense, and in fact it is not a problem. My question is how can I build short and inline inputs so that they are already evaluated before being passed to the scanner function. Right now, I can do this by restricting defaultMain to compare only one function at a time (as I just showed above), but this is not an ideal solution.
EDIT1
Something else is happening here with the Criterion criterion, and it seems to only happen in the Vector array, and not in the lists. If I force a full estimate by printing shortv and intv, the reference still measures the time as ~ 400 ns, not ~ 150 ns. Code update below:
main = do V.forM_ shortv $ \x -> do print x V.forM_ intv $ \x -> do print x defaultMain [ bench "encode ShortV" $ whnf encodeInt16V shortv ,bench "encode IntV" $ whnf encodeInt32V intv ]
Criterion output (also has deviations of 158.4%, which seem to be incorrect):
estimating clock resolution... mean is 5.121819 us (160001 iterations) found 253488 outliers among 159999 samples (158.4%) 126544 (79.1%) low severe 126944 (79.3%) high severe estimating cost of a clock call... mean is 47.45021 ns (35 iterations) found 5 outliers among 35 samples (14.3%) 2 (5.7%) high mild 3 (8.6%) high severe benchmarking encode ShortV mean: 382.1599 ns, lb 381.3501 ns, ub 383.0841 ns, ci 0.950 std dev: 4.409181 ns, lb 3.828800 ns, ub 5.216401 ns, ci 0.950 benchmarking encode IntV mean: 394.0517 ns, lb 392.4718 ns, ub 396.7014 ns, ci 0.950 std dev: 10.20773 ns, lb 7.101707 ns, ub 17.53715 ns, ci 0.950