Standard (simple?) Control code / test?

Question

Is there some kind of standard benchmarking system or plan or something else? I look at go, llvm, d and other languages, and I wanted to know how fair they are at runtime, memory usage, etc.

I found http://benchmarksgame.alioth.debian.org/ , but the code is not the same. One example is a C ++ source, 100 lines, and a C source 650. I hardly call it honest. Another test at its source has a dumb mistake, which is locking inside the loop, while other languages ​​expose it outside.

So, I wanted to know some tests that I could consider looking at / running, which may not use non-standard or even complex libraries. As implemented completely inside one source file. Something fair.

Answer 1

Benchmarking is not entirely honesty - it is the choice of something for your own workload in your limitations.

If you want to use the alioth shootout site, you can still get interesting information if you eliminate too complicated or too slow decisions (the exact balancing depends on what you want to do - write code that runs for five seconds or one that will occupy a dozen computers for five months). Look at the most concise examples for one specific problem to see the general structure of the problem, and then see what typical optimization users use to speed up code execution.

Having a reference with the EXACT code is not suitable, because you need different things to help in different languages; Java has GC, which means that it will work well with the trees test, while competing with it requires special allocation of memory in C / C ++ (and this particular test is structured so that the standard malloc does really bad), for spectral-norm one, you need double arrays without a box ...

If you want to come up with your own solutions, go to Project Euler - there are many problems that do not depend on complex libraries, but are difficult to optimize. Otherwise, try to find evaluation criteria that you think are appropriate for filtering or ranking existing contributions in the shootout (or outside it - for example, there are ShedSkin and Cython solutions for some of the problems that are "unofficial" because these languages ​​are not included).

Answer 2

For several years, the gaming site with the standards showed this on the help page -

What does "injustice" mean? (Fable)

They rushed up and down, and around, and around, and forward, and backward, and sideways, and upside down.

Cheetah's friends said “this is unfair” - everyone knows that the cheetah is the fastest creature, but the races are too long, and Gepa gets tired!

Falcon friends said “this is dishonest” - everyone knows that Falcon is the fastest creature, but Falcon does not go very well, it hovers through the sky!

Friends horses said “this is dishonest” - everyone knows that the horse is the fastest creature, but it's just a year old, you have to stop racing until the stallion takes part!

Male friends said “this is unfair” - everyone knows that in the “real world” a person will use a motorcycle, you have to wait until a person starts working and warms up the engine!

Street friends said "this is dishonest" - everyone knows that a creature must leave a trace of mucus, all these creatures are deceiving!

Dalmatian tail pounded on the ground. Dalmatian gasped and said between breaths: "Look at this beautiful mountain, let go of the race to the top!"

---

At that time, “dishonest” comments were mostly special pleas designed to take advantage of programming language X to the detriment of programming language Y.

But the problems that your question raises are slightly different.

• First, look at the n-body programs in the game Website. Although programs are written in different languages, there are very few differences in how programs are encoded.

  So far, no one has found an effective way to use quad-core processors for this small n-body problem - so there are no special multi-core programs there. Programs do not use non-standard or complex libraries. Programs are fully implemented within a single source code file.

• I said there is very little difference in how the n-body programs are encoded, but does this really mean that the same? Soon after the project was revived, 6 or 7 years ago, I remember the Ada programmer half joking about comparing apples to oranges, because the assembly language from Ada's programs was not the same as the assembly language of C programs - it’s so obvious that it wasn’t compared with such :-)

  • oto, Ada source code has to be written differently than C source code was written to make the Ada compiler the same assembly language as the C compiler is produced.

  • otoh, if the assembler language created by both compilers really were in turn the same, why the performance difference?

  When there are very few differences in the way the program is encoded, then at first glance the comparison seems fair, but forcing different languages ​​to encode, such as the X language, can contribute to the X language.

• As Yannick Versley noted, the point of using a different language is for the various approaches that the language provides. In other words, there is more than one way to do the same thing.

  Check out the mandelbrot programs on the benchmark gaming website — the simplest C program — half the size of the fastest C program; the simplest program C is sequential and uses doubles, the fastest program C uses all 4 cores through OMP and GCC built-in functions.

  • Other languages ​​use different approaches to using all 4 cores - does this mean that we should only compare sequential programs and ignore the reality of multi-core computing?

  • Other implementations of the language may not provide the equivalent of GCC intrinsics - does this mean that we should only compare programs that use doubling? But other implementations of the language use different approaches in the way they represent doubles - does this mean that we should ignore all floating point programs?

The problem is that the programming languages ​​(and the implementations of the programming language) are more different than apples for oranges, but we still ask: will my program be faster if I write it in X? - and still want a simpler answer than . It depends on how you write it!

Various tasks and various programs on the gaming machine website show that some of the answers for comparing performance are complex and complex - the details are important, many.