mbedtls icon indicating copy to clipboard operation
mbedtls copied to clipboard

Many tests are skipped in test-ref-configs

Open gilles-peskine-arm opened this issue 9 months ago • 2 comments

Looking at the logs of component_test_ref_configs on development, a lot of intended test cases are not executed. Contrast with 2.28. For example, with config-thread.h, which focuses on ECJPAKE, both in legacy and PSA mode:

 running ssl-opt.sh -f ECJPAKE.*nolog (config-thread.h)
 ECJPAKE: working, DTLS, nolog .......................................... SKIP
 ------------------------------------------------------------------------
 PASSED (1 / 1 tests (1 skipped))

This test case should pass, and does pass in 2.28.

A full log of all.sh test_ref_configs on 771fd7d1dcd19a157812dde9766bb7781e60d075, for convenience and to serve as a reference point: test-ref-configs-771fd7d1dcd19a157812dde9766bb7781e60d075.txt

The goal of this issue is to:

  • Figure out what's going wrong, and possibly since when.
  • Propose a method to determine whether the intended features are tested in each configuration.
  • If that method involves tooling, develop that tooling.
  • Fix all the bugs, i.e. arrange for each configuration's tests to actually run the intended tests.
  • Possibly design and implement a way to ensure this doesn't happen again.

This may be broken up into sub-issues depending on what the initial analysis finds.

gilles-peskine-arm avatar Apr 26 '24 16:04 gilles-peskine-arm

At least one of the problems is https://github.com/Mbed-TLS/mbedtls/issues/8366. But I think it's not the only one, I'll need to read the logs further.

gilles-peskine-arm avatar Apr 26 '24 18:04 gilles-peskine-arm

Possibly design and implement a way to ensure this doesn't happen again.

So far, we like the idea that ssl-opt.sh and compat.sh will, by default, complain if all test cases are skipped. (private discussion). This is far from perfect, but it's very easy to do, so it has a high efficacy/cost ratio.

gilles-peskine-arm avatar Apr 30 '24 11:04 gilles-peskine-arm