sys/benchmark: use turo
Open
kaspar030
opened this issue 4 years ago
•
19 comments
Contribution description
This changes sys/benchmark to use turo.
This enables the results to be collected by CI along with the other metrics.
For seamless use, sys/benchmark depends on the json output if no other turo output is selected .
The only users (tests/bench_runtime_coreapis and tests/periph_gpio) have their makefiles adopted to allow choosing the output format.
Testing procedure
Issues/PRs references
Currently depends on https://github.com/RIOT-OS/RIOT/pull/17027.
HiL Test Results
❌ frdm-k22f (2 fail test)
❌ nucleo-f091rc (2 fail test)
❌ nucleo-g474re (2 fail test)
❌ nucleo-f767zi (2 fail test)
❌ nucleo-l152re (2 fail test)
❌ nucleo-f207zg (2 fail test)
❌ nucleo-l432kc (2 fail test)
❌ samr34-xpro (1 fail test)
❌ saml10-xpro (1 fail test) (1 fail flash)
❌ saml11-xpro (1 fail test) (1 fail flash)
❌ samr30-xpro (2 fail test)
❌ samr21-xpro (2 fail test)
❌ nrf52dk (2 fail test)
❌ remote-revb (1 fail test) (1 fail flash)
❌ nucleo-l073rz (1 fail test)
❌ slstk3401a (2 fail test)
❌ nucleo-f411re (2 fail test)
❌ nucleo-f103rb (2 fail test)
❌ esp8266-esp-12x (2 fail test)
❌ esp32-wroom-32 (1 fail test)
❌ z1 (1 fail test) (1 fail flash)
❌ hifive1b (2 fail test)
❌ saml21-xpro (1 fail test)
❌ stk3200 (1 fail test) (1 fail flash)
😬 nucleo-f303re (1 fail flash)
😬 arduino-mega2560 (2 fail flash)
😬 frdm-kw41z (1 fail flash)
😬 arduino-due (1 fail flash)
😬 slstk3400a (1 fail flash)
😬 frdm-k64f (1 fail flash)
✅ same54-xpro
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
HiL Test Results
ah, yes, the tests need to be adapted I guess.
Please rebase
done, and I updated the test scripts of tests/periph_gpio and tests/bench_runtime_coreapis.
HiL Test Results
❌ slstk3401a (1 fail test)
😬 nucleo-g474re (1 fail flash)
😬 nucleo-f207zg (1 fail flash)
😬 arduino-mega2560 (2 fail flash)
😬 samr34-xpro (1 fail flash)
😬 frdm-k64f (1 fail flash)
😬 same54-xpro (1 fail flash)
✅ frdm-k22f
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ nucleo-f091rc
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ nucleo-f303re
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ nucleo-f767zi
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ nucleo-l152re
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ nucleo-l432kc
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ saml10-xpro
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ saml11-xpro
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ samr30-xpro
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ samr21-xpro
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ nrf52dk
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ remote-revb
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ nucleo-l073rz
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ nucleo-f411re
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ nucleo-f103rb
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ frdm-kw41z
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ esp8266-esp-12x
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ esp32-wroom-32
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ z1
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ hifive1b
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ arduino-due
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ slstk3400a
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ saml21-xpro
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
✅ stk3200
TEST
RESULT
tests/periph_gpio
✅ pass
tests/bench_runtime_coreapis
✅ pass
BTW, the pr also changes some types, like, for us/call is now a float, and per_sec is now a full uint64_t. This prevents some printing issues, like:
mutex_init(): 1us --- 0.000us per call --- 3567587328 calls per sec
This is tests/benchmark_runtime_coreapis on nrf52840dk with "CFLAGS_OPT=-O3", which essentially turns mutex_init() into a nop. 1us for 1000000 runs (which is bogus to start with) is not 3567587328 calls per sec, but 1000000000000.
This probably increases codesize of sys/benchmark's printing a bit.
nucleo-g474re (1 fail flash)
The output looks alright?
The only thing is the stk3200 that seems to look like the coms just messed up
nucleo-g474re (1 fail flash)
The output looks alright?
Ya it does... I will have to look into this (and re-enter my spiralling depression).
As it stands, flash failures are an infrastructure things and we should not worry about that.
Apart from the CI issues, does this look alright?
I also wouldn't mind if you piggy backed
diff --git a/sys/test_utils/result_output/txt/result_output_txt.c b/sys/test_utils/result_output/txt/result_output_txt.c
index ea4cddbed2..2bac84ff30 100644
--- a/sys/test_utils/result_output/txt/result_output_txt.c
+++ b/sys/test_utils/result_output/txt/result_output_txt.c
@@ -49,7 +49,7 @@ void turo_u64(turo_t *ctx, uint64_t val)
void turo_float(turo_t *ctx, float val)
{
(void)ctx;
- print_float(val, 8);
+ print_float(val, 7);
print_str(" ");
}
here
I also wouldn't mind if you piggy backed
Better not, who knows if @benpicco wants to backport those...
https://github.com/RIOT-OS/RIOT/pull/17036
Damn, this seems to enlarge tests by >3.5k.
Damn, this seems to enlarge tests by >3.5k.
Too many whitespaces :zany_face:?
Is it because of the old import stdio and fmt problem?
or the uint64_t and floats... I mean if they are needed than that is the cost.
or the uint64_t and floats... I mean if they are needed than that is the cost.
yeah, I'll see if I can drop those changes. Probably it's the float division and printing that pulls in >3k.
#17035 hast the milestone for 2022.04 set, but depends on this one. Does it seem realistic for you to include this for the upcoming release?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.
I'm not going to work on this anymore.
RIOT-OS
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}