Unlimited number of nominators eligible for Rewards payout

[Ank4n]

Related to https://github.com/paritytech/substrate/issues/12847

Changes

• ErasStakersClipped is phased out. Instead we split Exposure into multiple pages of Exposure with each page capped to the same size as ErasStakersClipped.
• The pages of exposure is not sorted anymore as we can guarantee all nominators are eligible to claim rewards.
• Introduction of ExposureOverview which has high level information about a validator's exposure for an era.
• StakingLedger.claimed_rewards is being phased out and renamed to StakingLedger.legacy_claimed_rewards. The claim history of paged rewards are stored in a new StorageDoubleMap ClaimedRewards.
• For older eras, the exposure is returned from ErasStakersClipped. This storage will become stale in 84 eras (HistoryDepth) and can be removed then.
• Rename MaxNominatorRewardedPerValidator to MaxNominatorRewardedPerPage.

Breaking changes

• Call payout_stakers accepts an extra parameter, page.

Description

Rewards payout is currently executed in a single block and limited to MaxNominatorRewardedPerValidator. This value is currently 512 on Kusama and 256 on Polkadot.

We can get rid of this limitation by paging the rewards payout, i.e. paying all the nominators backing a validator in multiple blocks. Each page size will then have some limitation, probably around the same value as MaxNominatorRewardedPerValidator today.

How payouts would look like after this change

Given an active validator on Kusama had 1100 backers for Era (E+1). In order to payout all the rewards, the caller (as this call is permissionless, it can be anyone but typically the validator) will need to call payout_stakers 3 times.

• payout_stakers(origin, stash, E+1, 0) => will pay the first 512 nominators.
• payout_stakers(origin, stash, E+1, 1) => will pay the second set of 512 nominators.
• payout_stakers(origin, stash, E+1, 2) => will pay the last set of 76 nominators. ...
• payout_stakers(origin, stash, E+1, 3) => will fail since this page does not exist.

Note: ErasStakersClipped used to store the top 512 nominators by sorting the nominator set first. With the paged reward payout, we do not need to sort this anymore. And so, the paged set of nominators are not guaranteed to be in the order of top contributor first.

Internal storage

Given the change goes to production at Era E, all exposure for given_era > E would be stored in a paged manner in ErasStakersPaged which is keyed by given_era, validator_stash and page. For given_era <= E, the exposure is stored in ErasStakersClipped for reward payout which can be thought of as the single paged exposure.

In 84 eras, all the entries in ErasStakersClipped would be stale as we only support up to $HistoryDepth (=84) eras of history. So after (E+84) eras, the old code can be cleaned up and we can drop ErasStakersClipped entirely.

TODO

• [ ] Should we expose via rpc the number of pages for a validator for an era? The UI should be able to calculate automatically by num_nominators/MaxNominatorRewardedPerPage.
• [ ] Tracker ticket for cleaning up the old code after 84 eras.
• [x] Weight calculation for pages which does not pay out validator.
• [ ] Consolidate EraInfo for all Era related logic.
• [ ] Add companion.
• [ ] Redo benchmarks in kusama and polkadot.

[Ank4n]

/cmd queue -c bench-bot $ pallet dev pallet_staking

[command-bot[bot]]

@Ank4n Could not find matching configuration bench-bot; available ones are bench, fmt, sample, try-runtime.

[Ank4n]

/cmd queue -c bench $ pallet dev

[command-bot[bot]]

@Ank4n https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304233 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment /cmd cancel 14-eb0bbc3d-10e7-4d47-82bb-eb724423f099 to cancel this command or /cmd cancel to cancel all commands in this pull request.

[command-bot[bot]]

@Ank4n Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304233 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304233/artifacts/download.

[Ank4n]

/cmd queue -c bench $ pallet dev pallet_staking

[command-bot[bot]]

@Ank4n https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304285 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment /cmd cancel 15-f21b7641-4887-4c97-8df5-1b7dcdf67cb5 to cancel this command or /cmd cancel to cancel all commands in this pull request.

[command-bot[bot]]

@Ank4n Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304285 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304285/artifacts/download.

[Ank4n]

/cmd queue -c bench $ pallet dev pallet_staking

[command-bot[bot]]

@Ank4n https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304805 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment /cmd cancel 16-83e2ce73-4ec3-4b4e-b7f6-1f24d6fd91e7 to cancel this command or /cmd cancel to cancel all commands in this pull request.

[Ank4n]

/cmd queue -c bench $ pallet dev pallet_staking

@try-runtime-bot

[command-bot[bot]]

@Ank4n https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304902 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment /cmd cancel 17-9ae88f00-f451-44c1-ba55-ae41443eae76 to cancel this command or /cmd cancel to cancel all commands in this pull request.

[command-bot[bot]]

@Ank4n Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304805 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304805/artifacts/download.

[command-bot[bot]]

@Ank4n Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304902 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2304902/artifacts/download.

[Ank4n]

/cmd queue -c bench $ pallet dev pallet_staking

[command-bot[bot]]

@Ank4n https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2305036 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment /cmd cancel 19-0b180d64-823c-4448-a427-3302d71ec1ba to cancel this command or /cmd cancel to cancel all commands in this pull request.

[command-bot[bot]]

@Ank4n Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2305036 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2305036/artifacts/download.

[Ank4n]

/cmd queue -c bench $ pallet dev pallet_staking

[command-bot[bot]]

@Ank4n https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2306257 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment /cmd cancel 20-dd14ed0d-6f74-4055-8995-b1f5489c3cb7 to cancel this command or /cmd cancel to cancel all commands in this pull request.

[command-bot[bot]]

@Ank4n Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2306257 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2306257/artifacts/download.

[kianenigma]

bot rebase

[paritytech-processbot[bot]]

Rebased

[ruseinov]

I'm going to dump this here, some comments that I have after the call:

1. Maybe mention in comments that sorting by balance is needed for clipping and that it's likely to be gone in the near future
2. I'd probably go for not having gov-adjustable constants to begin with, just do Get and bound vectors, see if there is a feature-request later for gov adjustments
3. Think about pallet versioning, it's pretty broken right now with most of pallets/crates having version 4.0.0-dev

[Ank4n]

bot bench $ pallet dev pallet_staking

[command-bot[bot]]

@Ank4n https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2431168 was started for your command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking. Check out https://gitlab.parity.io/parity/mirrors/substrate/-/pipelines?page=1&scope=all&username=group_605_bot to know what else is being executed currently.

Comment bot cancel 52-ff52a743-ddb4-4724-b638-d28192f1e805 to cancel this command or bot cancel to cancel all commands in this pull request.

[command-bot[bot]]

@Ank4n Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" pallet dev pallet_staking has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2431168 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2431168/artifacts/download.

[Ank4n]

bot bench $ runtime westend-dev pallet_staking

[command-bot[bot]]

@Ank4n "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" runtime westend-dev pallet_staking (https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2431633) was cancelled in https://github.com/paritytech/substrate/pull/13059#issuecomment-1440693121

[Ank4n]

bot cancel

[command-bot[bot]]

@Ank4n Command "$PIPELINE_SCRIPTS_DIR/commands/bench/bench.sh" runtime westend-dev pallet_staking has finished. Result: https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2431633 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/substrate/-/jobs/2431633/artifacts/download.