How can I compare code that modifies these settings?

Question

The current implementation of the built-in benchmarking tool runs the code inside the iter call several times each time the installation code is executed outside of iter . When the test code changes these settings, subsequent iterations of the compared code no longer compare the same thing.

As a specific example, I compare how quickly it takes to remove values ​​from Vec :

 #![feature(test)] extern crate test; use test::Bencher; #[bench] fn clearing_a_vector(b: &mut Bencher) { let mut things = vec![1]; b.iter(|| { assert!(!things.is_empty()); things.clear(); }); } 

This will fail:

 test clearing_a_vector ... thread 'main' panicked at 'assertion failed: !things.is_empty()', src/lib.rs:11 

Performing a similar test of a push element on a vector shows that iter closes almost 980 million times (depending on how fast this closure is). The results can be very misleading if there is one run that does what I expect and millions of others that do not.

Tests conducted with Rust nightly 1.19.0 (f89d8d184 2017-05-30)