MichaIng
Your 'benchmarking' sucks
https://forum.radxa.com/t/unofficial-dietpi-image-build-for-radxa-cubie-a5e/28400/9?u=tkaiser
Do you want this fail to be documented over at https://github.com/ThomasKaiser/sbc-bench/blob/master/Benchmarking_some_benchmarks.md or fix it within a reasonable time span?
Hello Thomas, it is always a pleasure to read your friendly productive issues 😄.
We do not officially support the Radxa Cubie A5E, and dietpi-benchmark is not meant to be super precise, but simple in the first place, to give some idea about performance differences between SBCs. If a more precise benchmark is needed, then there are surely better tools. But our benchmark should run through, and its result should be qualitatively correct at least.
I will have a look at it tomorrow, it is after midnight for me, and AFAIK also for you. Have a good night.
But our benchmark should run through, and its result should be qualitatively correct at least.
Absolutely not.
Extracting some productive points:
- Filtering out too fast outliers if the number of samples otherwise is large enough. Much too fast results are obviously falsely assigned hardware or faked. Not sure whether filtering too slow outliers makes sense as well, since possible resource utilization from other/background processes can affect all results the same way, cause also shifts within the expected range etc. We do already stop
dietpi-services-managed services, and applyperformancegovernor, to reduce disturbance, but of course we cannot rule disturbance out, and I don't know a way how to deal with it better, other than encouraging users to take care of it. - More based on my earlier ideas: Running tests for a fixed time rather than a fixed size/number of iterations. Especially on faster hardware, the tests are too short to flatten disturbance and start/stop impact. Numbers were like a magnitude or order slower, when this was once chosen 😅.
@ThomasKaiser Always a bundle of Joy. I see you are still spreading your love and intelligence with minimal effort using timeless humour and honor.
Back story: The original benchmark was made as a very simple and basic non-bulky way of obtaining a ROUGH idea of performance QUICKLY. Using Bash, no extras.
Accuracy: Correct, it is not and was never supposed to be on-par with "Official" benchmark tools.
End result: DietPi-Benchmark achieved its goal. But it could be better, like most things in life @ThomasKaiser. Although using threats without providing any solutions is somewhat pointless to all involved and achieves nothing really?
Issues: High core counts and using INT to split the load vastly increases the loss in accuracy. Benchmark finishes too quickly. It was designed years ago for a range of slower ARM’s using “A quick test” methodology, and, we had to ensure the RPi 1 didn't become the outlier and take weeks to finish.
Solution: I can only suggest @MichaIng make DietPi-Benchmark less “Sucks” and "Fails?" within a timespan that suits @ThomasKaiser's needs. Maybe @ThomasKaiser is up to the challenge, since he's the one that set it? 😉
I will leave this in your capable hands @MichaIng 😄 👍
