ssz icon indicating copy to clipboard operation
ssz copied to clipboard

Don't serialize immutable Validator data

Open dapplion opened this issue 3 years ago • 0 comments

Nimbus uses a very nice optimization that we should implement too:

When serializing state, don't serialize + deserialize Validator object fields withdrawal_credentials and pubkey. Instead store this data is a separate global append-only list.

  • Reduce disk size usage significantly
  • Reduce serialization + de-serialization time by +50%
  • Ensure data de-duplication when doing regen

To implement in SSZ, Container type must have some option to exclude certain fields from serialization. Then after reading the state, all validator object values must be iterated and mutated to append the immutable properties.

Nimbus references:

  • https://github.com/status-im/nimbus-eth2/blob/bef13b6cce244510ba989d3ce2a6176426592604/beacon_chain/beacon_chain_db_immutable.nim#L44
  • https://github.com/status-im/nimbus-eth2/blob/bef13b6cce244510ba989d3ce2a6176426592604/beacon_chain/spec/datatypes/base.nim#L282
  • https://github.com/status-im/nimbus-eth2/blob/bef13b6cce244510ba989d3ce2a6176426592604/beacon_chain/beacon_chain_db.nim#L514-L524

WIP branch

  • https://github.com/ChainSafe/ssz/pull/new/dapplion/container-omit-deserialize

dapplion avatar Jan 18 '22 19:01 dapplion