Beacon Chain Harness For Testing

[ensi321]

Problem description

Currently when it comes to testing the correctness of new changes that involve interacting with BeaconChain we rely on 1) unit tests 2) simulation tests 3) spec tests.

Unit tests require mocking up a chain, blocks and/or states. However since these objects contain many fields, one can only mock small parts of them that are related to the new changes. The nature of Ethereum protocol mandates validation in many places and a chain/block/state is only considered valid if the combination of the value in all fields pass the validation. Therefore passing unit tests with mocked chain/block/state does not guarantee the changes will work with a valid chain/block/state. Lodestar provides MockedBeaconChain but it only provides a chain that is TS type check correct and not protocol correct.

Simulation test creates an ephemeral devnet to test the overall functionality of Lodestar. It might be an overkill to test a small change. Also its error outputs might not be useful to tell which part of the new change is wrong. It is also resource intensive that is not designed to run on local machines for sanity checks but on the CIs.

Spec tests cover the correctness of the consensus protocol only. Things like caches, regen, Beacon api also interact with chain/block/state but not part of the protocol.

It is not hard to see the benefit of the ability to easily create a valid and functioning chain/block/state that is interact-able during unit tests.

Solution description

Create a suite of test utilities that can

• Create a valid beacon chain that can
  • Process blocks
  • Create a valid genesis state
  • Able to serve reasonably according to IBeaconChain interface
• Create a valid beacon block that can
  • Go through state transition
  • Be created with desired properties
    • Contains proposer slashing
    • Missed attestations from some validators
    • Split view on head block
    • Absent or present of sync committee
    • etc.
• Create valid unaggregated attestations
• Create valid sync committee messages
• Create valid slashings
• Create valid withdrawals/deposits
• Create valid bls to execution change
• Dial state forward to certain slot

The test suite in general should be:

• Lightweight
• Verbal on logs
• Can be well inspected

Additional context

Lighthouse's harness for reference: https://github.com/sigp/lighthouse/blob/b5bae6e7a2f52310617a3b93ab7579f3dc9d7880/beacon_node/beacon_chain/src/test_utils.rs#L604

[wemeetagain]

Really like the idea. I know I've personally wanted this for testing and would find it useful.

[g11tech]

we have dev network/beacon/validator client creation which does this (and is how merge-interop tests spinup beacon node)

what is it do you think from the above mentioned points that is lacking there? would love to see how we can make it more powerful / complete

[twoeths]

yes it'll produce something similar to dev network but we don't need to set up validators/beacon chain, we want test utils to be consumed by different unit/e2e tests

[jeluard]

@ensi321 If you had to name / describe those tests similarly to what is done in the testing page, what would that be?

It could also be an opportunity to write some guidelines detailing which tests are expected to be added depending on the feature being introduced.

[wemeetagain]

@ensi321 If you had to name / describe those tests similarly to what is done in the testing page, what would that be?

It could also be an opportunity to write some guidelines detailing which tests are expected to be added depending on the feature being introduced.

Looks like this would facilitate integration testing. The 'feature' is testing infrastructure that lets us use realistic data rather than stubs.

[ensi321]

@ensi321 If you had to name / describe those tests similarly to what is done in the testing page, what would that be?

Agree with what @wemeetagain said. Should be integration testing. But realistically, this new test utils can be used in various places such as packages/*/test/unit to ensure we have valid beacon chain related objects/containers to test with.

It could also be an opportunity to write some guidelines detailing which tests are expected to be added depending on the feature being introduced.

Oh that's a good point.

Off the top of my head:

• unit test should be written whenever we add/update features.
• spec test should be updated whenever we have changes related to consensus spec implementation (new EIP, new fork etc.)
• the bigger tests should be untouched most of the time.

We can discuss this elsewhere

[philknows]

Tagging as help-wanted to signal as a potential project for an EPF candidate.