ic icon indicating copy to clipboard operation
ic copied to clipboard

fix(NNS): Fix flaky node_provider_remuneration test

Open pietrodimarco-dfinity opened this issue 4 weeks ago • 0 comments

Fix test flakiness in test_automated_node_provider_remuneration

Problem The test_automated_node_provider_remuneration test was experiencing flaky failures with TLS certificate validation errors:

Error from Canister rwlgt-iiaaa-aaaaa-aaaaa-cai: Canister called `ic0.trap` with message: '[Registry] Add node failed: [Registry] do_add_node: Could not validate public keys, due to KeyValidationError { error: "invalid TLS certificate: notBefore date (=ASN1Time(2025-12-18 15:07:50.0 +00:00:00)) is in the future compared to current time (=ASN1Time(2025-12-18 15:07:19.0 +00:00:00))" }'

This flakiness occurred because the test relied on an implicit timing assumption that could fail depending on when the test started and how quickly operations executed.

Root Cause The flakiness stems from a timing race condition between:

  1. Certificate generation (in gen_tls_key_pair_internal): Uses SystemTime::now() from the system's wall clock, then subtracts 2 minutes to set the certificate's notBefore date.
  2. Certificate validation (in the Registry canister): Uses the StateMachine's internal time, which starts at a default epoch time and doesn't automatically sync with real-world time.

When certificates are generated during prepare_add_node_payload(), they use the current system time. However, if the state machine's time hasn't been explicitly set or has fallen behind, certificate validation fails because the state machine believes it's checking a certificate "from the future." This creates a race condition where the test could pass or fail depending on timing factors outside the test's control.

Solution Explicitly synchronize the state machine's time to the current system time (SystemTime::now()) before adding nodes. This eliminates the race condition by ensuring the state machine's time is always recent enough to validate freshly-generated certificates.

pietrodimarco-dfinity avatar Dec 18 '25 16:12 pietrodimarco-dfinity