qbft test fails intermittently on the CI workflow

[boulder225]

🎯 Problem to be solved

The qbft test fails intermittently on the CI workflow but passes locally. This discrepancy is due to the difference in the behaviour of the test across environments, which indicates a potential timing issue or environmental condition.

FAIL	github.com/obolnetwork/charon/core/qbft	1.635s
ok  	github.com/obolnetwork/charon/core/scheduler	2.433s	coverage: 85.3% of statements
ok  	github.com/obolnetwork/charon/core/sigagg	2.144s	coverage: 81.1% of statements
ok  	github.com/obolnetwork/charon/core/tracker	1.350s	coverage: 73.1% of statements
ok  	github.com/obolnetwork/charon/core/validatorapi	2.560s	coverage: 76.2% of statements
?   	github.com/obolnetwork/charon/testutil	[no test files]
?   	github.com/obolnetwork/charon/testutil/compose/compose	[no test files]
?   	github.com/obolnetwork/charon/testutil/promrated/promrated	[no test files]
ok  	github.com/obolnetwork/charon/dkg	54.808s	coverage: 81.7% of statements
ok  	github.com/obolnetwork/charon/dkg/bcast	2.172s	coverage: 85.4% of statements
ok  	github.com/obolnetwork/charon/dkg/sync	5.564s	coverage: 82.0% of statements
ok  	github.com/obolnetwork/charon/eth2util	1.498s	coverage: 43.6% of statements
ok  	github.com/obolnetwork/charon/eth2util/deposit	1.077s	coverage: 77.8% of statements
ok  	github.com/obolnetwork/charon/eth2util/eip712	1.015s	coverage: 86.4% of statements
ok  	github.com/obolnetwork/charon/eth2util/enr	3.078s	coverage: 84.5% of statements
ok  	github.com/obolnetwork/charon/eth2util/eth2exp	1.076s	coverage: 19.4% of statements
ok  	github.com/obolnetwork/charon/eth2util/keymanager	1.045s	coverage: 86.5% of statements
ok  	github.com/obolnetwork/charon/eth2util/keystore	11.770s	coverage: 78.3% of statements
ok  	github.com/obolnetwork/charon/eth2util/registration	1.085s	coverage: 80.0% of statements
ok  	github.com/obolnetwork/charon/eth2util/rlp	1.033s	coverage: 82.1% of statements
ok  	github.com/obolnetwork/charon/eth2util/signing	1.207s	coverage: 60.0% of statements
ok  	github.com/obolnetwork/charon/p2p	4.735s	coverage: 36.2% of statements
ok  	github.com/obolnetwork/charon/tbls	1.128s	coverage: 58.2% of statements
ok  	github.com/obolnetwork/charon/tbls/tblsconv	1.027s	coverage: 94.1% of statements
ok  	github.com/obolnetwork/charon/testutil/beaconmock	1.297s	coverage: 46.4% of statements
ok  	github.com/obolnetwork/charon/testutil/compose	1.441s	coverage: 35.2% of statements
ok  	github.com/obolnetwork/charon/testutil/compose/fuzz	1.029s	coverage: [no statements]
ok  	github.com/obolnetwork/charon/testutil/compose/smoke	1.028s	coverage: [no statements]
ok  	github.com/obolnetwork/charon/testutil/genchangelog	1.029s	coverage: 16.9% of statements
ok  	github.com/obolnetwork/charon/testutil/integration	1.099s	coverage: [no statements]
ok  	github.com/obolnetwork/charon/testutil/promrated	1.077s	coverage: 45.6% of statements
ok  	github.com/obolnetwork/charon/testutil/validatormock	1.089s	coverage: 30.7% of statements
ok  	github.com/obolnetwork/charon/testutil/verifypr	1.025s	coverage: 67.0% of statements
FAIL
Error: Process completed with exit code 1.

🛠️ Proposed solution

• [ ] Investigate the CI environment's load and performance characteristics
• [ ] Compare the execution environment and configuration between local and CI
• [ ] Adjust the covermode=atomic setting
• [ ] Document the findings and update the testing guidelines
• [ ] Implement a monitoring solution to watch for these failures