Some V2 tests are flaky

[goatgoose]

Problem:

Some V2 tests are flaky, and cause problems when running each test multiple times in different build environments, since the flakiness is exaggerated. The tests that appear to be the most flaky are the following:

• test_early_data -> test_s2n_server_with_early_data_rejected
  • Codebuild run: https://us-west-2.console.aws.amazon.com/codesuite/codebuild/024603541914/projects/S2nIntegrationV2Batch2/build/S2nIntegrationV2Batch2%3A1ea45135-35f9-4365-979e-2953ec907b83/log?region=us-west-2
• test_key_update -> test_s2n_server_key_update
  • Codebuild run: https://us-west-2.console.aws.amazon.com/codesuite/codebuild/024603541914/projects/S2nIntegrationV2Batch2/build/S2nIntegrationV2Batch2%3A84854ab0-b538-40e5-9baa-950c1cc0a822/log?region=us-west-2

The libcrypto that s2n-tls was built with may have some impact on the flakiness, since the test_key_update run in the previous example failed 3 tests in the same run with awslc-fips.

Temporarily, the rerunfailures pytest extension was added that retries failed tests without needing to restart an entire Codebuild run.

Solution:

Investigate why these tests are flaky, and fix them. If the tests become sufficiently reliable, the rerunfailures extension can be removed.

• Does this change what S2N sends over the wire? If yes, explain.
• Does this change any public APIs? If yes, explain.
• Which versions of TLS will this impact?

Requirements / Acceptance Criteria:

The V2 tests are less flaky.

• RFC links: Links to relevant RFC(s)
• Related Issues: Link any relevant issues
• Will the Usage Guide or other documentation need to be updated?
• Testing: How will this change be tested? Call out new integration tests, functional tests, or particularly interesting/important unit tests.
  • Will this change trigger SAW changes? Changes to the state machine, the s2n_handshake_io code that controls state transitions, the DRBG, or the corking/uncorking logic could trigger SAW failures.
  • Should this change be fuzz tested? Will it handle untrusted input? Create a separate issue to track the fuzzing work.

Out of scope:

N/A