realtime testing suite evaluation

[acerv]

LTP has a realtime testing suite that is implemented into realtime folder and it looks more like an extension of the LTP framework. For example, we don't have any runtest file defining a proper realtime testing suite.

This ticket aims to remember we could probably start a good analysis on what we have for realtime testing and add it to the runtime internals.

[pevik]

We already got some analysis long time ago from Mel Gorman (asking privately).

TL;DR:

1. port pi_perf, prio-wake to rt-tests.
2. ask the community for opinion on porting sched_football (sched_football would need to be reimplemented) https://git.kernel.org/pub/scm/utils/rt-tests/rt-tests.git

Note, Mel only evaluated content of https://github.com/linux-test-project/ltp/tree/master/testcases/realtime/func, i.e. not evaluated https://github.com/linux-test-project/ltp/tree/master/testcases/realtime/stress/pi-tests and https://github.com/linux-test-project/ltp/tree/master/testcases/realtime/perf/latency.

Mel Gorman:

It would make more sense to port anything useful in there to rt-tests that the preempt_rt community pay attention to. That said, Daniel is much more familiar with the community than I am and may correct me.

I glanced through the LTP tests. A major weakness is the reporting of PASS/FAIL. Some are based on static values that are potentially machine-specific.

• async_handler Potentially interesting but if it fails completely, it still reports PASS. Otherwise, I think it can be replaced with perf bench futex combined with perf sched latency.
• hrtimer-prio Very similar to rt-tests deadline_test
• gtod_latency Measures the latency of clock_gettime. I'm not sure how useful this really is. Fair enough, it's important but an rt app is unlikely to call it at high frequency but it'll depend on the clock source and the results are vulnerable to scheduler artifacts
• matrix_mult This is basically a CPU performance test. There are other ways that can be done and is not RT specific
• measurement The preempt_timing timing test can be replaced with cyclictest from rt-tests. The rdtsc-latency test is borderline useless. rdtsc should almost never be used as a time source
• periodic_cpu_load Potentially interesting but the static values would make it difficult to calibrate.
• pi_perf Very interesting, might be worth porting to rt-tests
• prio-preempt Think this could be replaced with rt-migrate-test in rt-tests
• prio-wake Very interesting, might be worth porting to rt-tests
• pthread_kill_latency Not sure this is very useful. A more useful check is wakeup latency which can be done with the perf tool -- perf bench futex
• rt-migrate-test Already ported to rt-tests
• sched_football Interesting, might be worth reimplementing. The concept is interesting as a test but it has weaknesses. The sched_yield usage is one thing but also, it only works if the number of players equals the number of CPUs and small changes in sched_yield semantics would break the test.
• sched_jitter sysjitter is a replacement
• sched_latency A more appropriate way to do this is running arbitrary workloads under perf sched latency
• thread_clock Vaguely interesting as a functional test but not high priority

So from the list, I think the only ones of interest are pi_perf, prio-wake and maybe sched_football although sched_football would need to be reimplemented.

[pevik]

Cyril considered sched_football useful enough also for regular non-RT systems, thus I ported it to new library in 43dc52fafa60a9eace9ee8d81576117736cfb5e8.

And there is a followup patchset which among other things always enables realtime and put sched_football also to runtest/sched: https://patchwork.ozlabs.org/project/ltp/list/?series=415662&state=* https://lore.kernel.org/ltp/[email protected]/

But we should decide what to do with the rest.

[pevik]

And there is a followup patchset which among other things always enables realtime and put sched_football also to runtest/sched: https://patchwork.ozlabs.org/project/ltp/list/?series=415662&state=* https://lore.kernel.org/ltp/[email protected]/

Patchset merged (realtime related commits: 149258619842fdfd8dcf6243466791807dcda12b, 2f040ad8352a772f12a72049d49a28f35222e3ee, 3a8e38fc2c46c886525bbb708dff462c6b633d32), but 3a8e38fc2c46c886525bbb708dff462c6b633d32 caused additional problem, because it's installed into non-standard directory $LTPROOT/testcases/realtime/func/sched_football/). Suggested solutions: https://lore.kernel.org/ltp/20240729073226.GA1223191@pevik/.

[zjylikesmile]

Could you please advise on how we should run the realtime testing framework? According to the latest tag, the README instructs to run run.sh, but I found that this script is not present in the actual build.

[metan-ucw]

Looks like the run.sh script is not installed, I will send a patch to the ML.