fortran-lang
Add Benchmarks sections
I suggest that we have to have and maintain our own benchmarks and have a dedicated section for it.
As an example, the Julia page has benchmarks against Fortran, but unfortunately they do not enable all optimization flags in gfortran, and when I pointed that to them, they said that they do not want to do that because they have different priorities in what they want to benchmark: https://github.com/JuliaLang/julia/issues/24568, which is fine, they have the right to do that. But it is then misleading to users when they think that Fortran is not that fast.
To fix that, we have to maintain our own benchmarks, and test them with several Fortran compilers, and ensure that all optimization options are enabled. And we should benchmark against other languages.
This will also provide motivation for Fortran compiler vendors to speed up some of these benchmarks if Fortran is not the fastest.
Here are some ideas for benchmarks to include:
- https://julialang.org/benchmarks/
- https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/fortran.html
- https://github.com/ParRes/Kernels ...
Furthermore, I suggest to set objectives of what code we will accept:
- Must only use the given language, for example Fortran or C++, and it should not use any assembly directly.
As an example, for the n-body problem, this code is fine for Fortran: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-ifc-6.html, readable, maintainable. For C++, we should include an equivalent code, we specifically should not allow to include assembly instrinsics like this one: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-gpp-7.html, full of assembly intrinsics. More specifically, we should mention it that you can of course write this in assembly (or inline it in C++ in various ways), but in our main benchmarks, we should put code that people would actually write to solve the problem in the given language itself without resolving to assembly. The same with the Rust version that currently is the fastest: https://benchmarksgame-team.pages.debian.net/benchmarksgame/program/nbody-rust-7.html, it's full of assembly intrinsics.
Then we can have a section where anything is allowed, including assembly.
The idea is that with enough expertise, the best is to optimize in assembly. For example OpenBLAS is mostly in assembly. But the point of the benchmarks is to see how fast can you get if you only stick to the language and canonical ways of doing things in the language.
To put it another way: our benchmarks should show, that Fortran is an excellent choice for this mission:
- enable scientists, engineers, and other domain experts to write programs that naturally express the mathematics and algorithms employed, are portable across HPC systems, remain viable over decades of use, and extract a high percentage of performance from the underlying hardware.
Yes, it's paramount that we set the objectives ahead of time so we're clear ourselves what is it that we want to measure. Considering the scope of the project, I think this issue calls for a dedicated repo, fortran-lang/benchmarks.
Yes, definitely a dedicated repository, so that people can easily run the benchmarks on their systems. Only the results can be shown at fortran-lang.org.
+1 for a benchmarks repo!
Ideally, for the web-published results we would have a dedicated machine for automation and consistency of test-environment, but I guess this isn't strictly necessary if we're interested in relative comparisons.
Do we only test free compilers like gfortran and flang or can we get a license for ifort and others?
This will also provide motivation for Fortran compiler vendors to speed up some of these benchmarks if Fortran is not the fastest.
I agree, it will also identify any areas where Fortran is lagging behind; in the Julia benchmarks, Fortran seems to be let down by the integer parsing and the file io benchmarks (implementation here).
The Debian shootout page is using Intel Fortran, so I suggest we use all Fortran compilers that are available to us. In fact, this can be a nice opportunity to compare the various Fortran compilers themselves. We can have different sections in the benchmarks, such as:
- Fortran only (Fortran compilers comparison)
- Languages comparison (the best is probably to show various Fortran and C++ and other languages' compilers); inline assembly and intrinsics are not allowed
- Assembly section (where any code in any form is allowed)
The "Languages comparison" section can also have subsections, like
- all optimizations on (the default section)
-ffast-mathnot allowed (that's essentially the Julia benchmarks page), which has its merits, but I would argue most Fortran users care more about the "all optimizations on" section
See also a related thread at https://fortran-lang.discourse.group/t/improving-fortran-results-in-the-julia-micro-benchmarks/198.
Let's start the repository.
How about https://github.com/fortran-lang/benchmarks?
Sounds good! We can then discuss various options for implementation in the dedicated repo.
Done, the repository is up, @LKedward you should have maintainer access. Go ahead and start it.