Below is an alphabetical list of all the tools I found. Described Aspects:
- is it easy to parameterize
- it’s a Java library, or at least easily integrated into your Java program.
- can it handle micro benchmarking JVM, for example. use warm-up phase
- you can visually display the results
- measured values can be stored continuously
- can he analyze the trend to warn that a new commit has caused a slowdown.
- provides and uses statistics (at least maximum, minimum, average and standard deviation).
Autopilot
parameterizable; Perl Library no micro benchmarking JVM; drawing; perseverance; trend analysis!?; good statistics (run this test until the results are stable, highlight the outliers).
Basic scheme
not parameterized; Java library Microprocessor JVM; no schedule; no perseverance; no trend analysis; statistics.
Is the statistics good: in addition to the average, maximum, minimum, and standard deviation, it also calculates a 95% confidence interval (via download) and a consistent correlation (for example, to warn about oscillatory runtimes that may occur if your program behaves non-deterministically, for example because you use HashSets). He decides how often the program needs to be repeated to get accurate measurements and interprets them for reports and warnings (for example, emissions and consistent correlation).
Micro benchmarking also works very well (see Create a fast / reliable test with java? ).
Unfortunately, the structure is part of the util package, which comes with many other helper classes. Test levels depend on JSci (API for Java) and Mersenne Twister ( http://www.cs.gmu.edu/~sean/research/ ). If the author, Brent Boyer, finds the time, he will boil the library and add a simpler graph so that the user can visually check the measurements, for example. for correlations and outliers.
Caliper
parameterizable; Java library Microprocessor JVM; drawing; perseverance; no trend analysis; statistics.
A relatively new project focused on Android applications. It looks young but promising. Depends on Google Guava :(
Community monitoring
not parameterized!?; Java library no micro benchmarking JVM!?; drawing; persistence through the servlet; no trend analysis!?; no statistics!?.
Supports AOP hardware.
Jamon
not parameterized; Java library no micro benchmarking JVM; plotting, saving and analyzing trends using additional tools (Jarep or JMX); statistics.
Good monitoring, interwoven with log4j, can also be programmed or requested programmatically, and your program can take action on the results.
Java simon
not parameterized!?; Java library no micro benchmarking JVM; Build only with Jarep; persistence only with JMX; no trend analysis; no statistics!?.
Member Jamon, maintains a hierarchy of monitors.
Jetm
not parameterized; Java library Microprocessor JVM; drawing; perseverance; no trend analysis; no statistics.
A good easy monitoring tool, without dependencies :) It does not provide sufficient statistics (without standard deviation), and the plugin extension looks rather complicated accordingly (aggregators and aggregates have fixed getters for minimum, maximum and average).
jmeter
parameterizable!?; java library; no micro benchmarking JVM!?; drawing; perseverance; trend analysis!?; statistics!?.
A good monitoring library designed for load-loaded web applications.
Java Microbenchmark Harness ( jmh )
parametrizable (custom invokers via the Java API); Java library JVM Micro Lens; no sites; no perseverance; no trend analysis; statistics.
Hard benchmarking, built by Oracle HotSpot experts, which is very suitable for micro-capture on HotSpot, used in OpenJDK. Extraordinary measures are being taken to provide a reliable benchmarking environment. Besides readable output, jmh provides a Java API for processing results, for example. for third-party conspirators and persistence providers.
junit-benchmarks
parameterizable; Java library Microprocessor JVM; drawing; persistence (using CONSOLE, XML, or H2 database); graphical trend analysis; statistics (max, min, average, standard deviation, but not easily extensible for further statistics).
Just add the junit-4 rule to your junit tests :)
junit-Benchmarks is open source under the Apache 2 license.
Update : project moved to jmh
junitperf
Mainly for analyzing performance trends (using JUnit decor decorator TimedTest ) and scalability (using JUnit decor decorator LoadTest ).
parameterizable; Java library no micro benchmarking JVM; no schedule; no perseverance; no statistics.
perf4j
not parameterized; Java library no micro benchmarking JVM; drawing; persistence through JMX; trending through the log4j app; statistics.
Creates a logging structure, can use AOP.
Project broadway
A very general concept: observers observe predefined conditions and determine how to respond when they are fulfilled.
speedy-mcbenchmark
The focus is on parametrizability: check if your algorithm is scaled, i.e. does it check O (n), O (n log (n)), O (n²) ...
java library; Microprocessor JVM; no schedule; perseverance; trend analysis; no statistics.
The grinder
parameterizable; Jython Library no micro benchmarking JVM; drawing; perseverance; no trend analysis; no good statistics, but easily extensible.
Depends on Jython, HTTPClient, JEditSyntax, ApacheXMLBeans, PicoContainer.
TPTP
parameterizable!?; Java tool platform no micro benchmarking JVM!?; drawing; perseverance; graphical trend analysis; no statistics !?
The test and performance platform is a huge, versatile and extensible tool platform (based on Eclipse and four EMF models). Therefore, it is powerful, but rather complex, it can slow down Eclipse and expand it for your own needs (for example, with statistics, so that they affect the number of iterations), it seems very difficult.
Usemon
parameterizable!?; Java library no micro benchmarking JVM; drawing; perseverance; trend analysis!?; statistics!?.
The tool is intended for monitoring in large clusters.