Analyze missing unit test coverage for normal TLS connections

[gilles-peskine-arm]

Ideally all the code in Mbed TLS should be covered by unit tests. We know that this is not the case for TLS, where testing relies on system tests (ssl-opt.sh, compat.sh). The goal of this task is to analyze the missing test coverage of unit tests relative to compat.sh.

1. Measure the test coverage in the full configuration from unit tests plus compat.sh. Test coverage measurement is defined as in our code-coverage CI job. This is tests/scripts/basic-build-test.sh minus ssl-opt.sh.
2. Repeat with just the unit tests, i.e. tests/scripts/basic-build-test.sh minus ssl-opt.sh and compat.sh.
3. Post the reports here.
4. File issues to add unit tests for the missing coverage: when those issues are resolved, compat.sh should be useless in the code-coverage job, enabling #5347.

[zhangsenWang]

The test reports are listed below. And the detailed logs are attached.

1. unit tests plus compat.sh:

Unit tests - tests/scripts/run-test-suites.pl
No test suites     : 100
Passed             : 16612
Failed             : 0
Skipped            : 8905
Total exec'd tests : 16612
Total avail tests  : 25517

System/Compatibility tests - tests/compat.sh
Passed             : 680
Failed             : 48
Skipped            : 2
Total exec'd tests : 728
Total avail tests  : 730

-------------------------------------------------------------------------
Total tests
Total Passed       : 17292
Total Failed       : 48
Total Skipped      : 8907
Total exec'd tests : 17340
Total avail tests  : 26247

2. only unit tests

Unit tests - tests/scripts/run-test-suites.pl
No test suites     : 100
Passed             : 16612
Failed             : 0
Skipped            : 8905
Total exec'd tests : 16612
Total avail tests  : 25517

-------------------------------------------------------------------------
Total tests
Total Passed       : 16612
Total Failed       : 0
Total Skipped      : 8905
Total exec'd tests : 16612
Total avail tests  : 25517

only_unit.log unit_compat.log

[zhangsenWang]

File issues to add unit tests for the missing coverage: when those issues are resolved

@gilles-peskine-arm Do you mean that all the tests in compat.sh should be packed into unit suit test like those in tests/suites/ and be used by scripts/run-test-suites.pl ? It seems that the test in compat.sh differs much with the tests in tests/suites/. So what can be called as a unit with the tests in compat.sh ? one test case one unit ?

[gilles-peskine-arm]

Do you mean that all the tests in compat.sh should be packed into unit suit test like those in tests/suites/ and be used by scripts/run-test-suites.pl ?

No. I don't expect compat.sh to change.

The primary goal of compat.sh is to test interoperability. If we only had unit tests, and a bug in TLS code that's common to the client and the server, it's likely the unit tests would still pass. So we test an mbedtls client with an independently written server and vice versa.

But discovering bugs in compat.sh is a bit late, we don't run it in all configurations, it's annoying to debug. So it's better if we have as much coverage as possible from unit tests.

The goal of this task is to identify parts of the code that are not tested by unit tests. The goal of follow-ups will be to add unit tests for those parts of the code.

So the first step is to run what you did, but get the detailed report (not just the statistics) of lines/functions/branches that are covered by compat.sh but not by unit tests. The next step is to analyse this detailed report to determine what the unit tests should do.

[gilles-peskine-arm]

As discussed on Slack, analyzing the differences is likely to be difficult. Even if we know that a particular line/branch isn't executed by the unit tests, it's not necessarily straightforward to do figure out what a good unit test for it would be.

We can conduct an SSL handshake in a unit test. While we may want to have additional, more specific unit tests, having complete handshakes as unit tests would be a good start.

I'm going to make a task breakdown for creating unit tests that essentially replicate compat.sh -p mbedTLS. Once that's done, the missing test coverage should be a lot less — only things that we missed in the new unit tests, or bits of code that happen to be invoked in an mbedtls client/server only with certain servers/clients that aren't mbedtls.

So, @zhangsenWang, please keep what you've already done to set up the environment and please commit any scripts you've made to automate the comparison. But we'll pause the analysis until we've done the direct replication unit tests.