mbedtls
mbedtls copied to clipboard
Many tests are skipped in test-ref-configs
Looking at the logs of component_test_ref_configs
on development
, a lot of intended test cases are not executed. Contrast with 2.28. For example, with config-thread.h
, which focuses on ECJPAKE, both in legacy and PSA mode:
running ssl-opt.sh -f ECJPAKE.*nolog (config-thread.h)
ECJPAKE: working, DTLS, nolog .......................................... SKIP
------------------------------------------------------------------------
PASSED (1 / 1 tests (1 skipped))
This test case should pass, and does pass in 2.28.
A full log of all.sh test_ref_configs
on 771fd7d1dcd19a157812dde9766bb7781e60d075, for convenience and to serve as a reference point: test-ref-configs-771fd7d1dcd19a157812dde9766bb7781e60d075.txt
The goal of this issue is to:
- Figure out what's going wrong, and possibly since when.
- Propose a method to determine whether the intended features are tested in each configuration.
- If that method involves tooling, develop that tooling.
- Fix all the bugs, i.e. arrange for each configuration's tests to actually run the intended tests.
- Possibly design and implement a way to ensure this doesn't happen again.
This may be broken up into sub-issues depending on what the initial analysis finds.
At least one of the problems is https://github.com/Mbed-TLS/mbedtls/issues/8366. But I think it's not the only one, I'll need to read the logs further.
Possibly design and implement a way to ensure this doesn't happen again.
So far, we like the idea that ssl-opt.sh
and compat.sh
will, by default, complain if all test cases are skipped. (private discussion). This is far from perfect, but it's very easy to do, so it has a high efficacy/cost ratio.