Fix staking collator snapshot to use total counted instead of total exposure

[4meta5]

When paying out rewards, we calculate delegation reward payout in proportion to delegation_total / total_backing_collator. The denominator should be the sum of all counted delegations instead of the the sum of all exposed delegations.

By counting the delegation amounts that are excluded (because pending request) in the denominator, we are underestimating the reward payouts for all delegations and the collator itself whenever there are pending requests. The more pending requests for a collator's delegations, the more rewards are underpaid to the collator and its delegations.

The unit tests diff shows this.

Adds 2 smoke tests:

1. for this specific bug by checking CollatorSnapshot and ensuring bond + delegations = total
2. verify inflation by checking sum of rewarded event.amount for round == totalRoundIssuance - parachainBondPayout

[crystalin]

Don't we have a smoke test for this ? If not please add one

[4meta5]

Don't we have a smoke test for this ? If not please add one

Adding it now. The smoke test will check that the CollatorSnapshot.total == CollatorSnapshot.delegations.sum() + CollatorSnapshot.self_bond (this will fail on all networks as of right now because of the bug explained in the PR description).

[4meta5]

I'm having trouble running the smoke tests that are updated in this PR cc @nbaztec Here is the command:

$ DEBUG=smoke:test-staking-rewards WSS_URL=wss://moonbeam-alpha.api.onfinality.io/public-ws npm run smoke-test

I've tried all the public moonbase endpoints.

I'm getting this error from most tests: Error: Endpoint should start with 'ws://', received 'null'

The other error I'm getting is TypeError: Cannot read properties of undefined (reading 'disconnect')

[nbaztec]

I'm having trouble running the smoke tests that are updated in this PR cc @nbaztec Here is the command:

$ DEBUG=smoke:test-staking-rewards WSS_URL=wss://moonbeam-alpha.api.onfinality.io/public-ws npm run smoke-test

I've tried all the public moonbase endpoints.

I'm getting this error from most tests: Error: Endpoint should start with 'ws://', received 'null'

The other error I'm getting is TypeError: Cannot read properties of undefined (reading 'disconnect')

You also need the RELAY_WSS_URL for the network - that's probably what's missing. I'll improve on that error.

RELAY_WSS_URL=wss://frag-moonbase-relay-rpc-ws.g.moonbase.moonbeam.network WSS_URL=wss://wss.api.moonbase.moonbeam.network npm run smoke-test

[4meta5]

Thanks @nbaztec , that was it!

Output shows at least 1 instance of the bug on moonbase, probably more exist but purpose is not to collect them all, just to report when it happens:

1) Verify staking rewards
       rewards are given as expected:

      Total counted (denominator) - total counted (numerator) = 15130000000000000000 so this collator and its delegations receive fewer rewards for round 4366
      + expected - actual

      -23922170000000000000000
      +23907040000000000000000

[4meta5]

• [x] consistency test that total rewards (all blocks in previous round) are within inflation range

[4meta5]

Newest smoke test output:

1) Verify staking rewards
       rewards are given as expected:

      AssertionError: Total rewarded events did not match total expected issuance for collators + delegators:
    72911764224129632453 != 491634853852524770177 

    Inflation was 418723089628395137724 less than expected for round 4407: expected '72911764224129632453' to equal '491634853852524770177'
      + expected - actual

      -72911764224129632453
      +491634853852524770177
      
      at assertRewardsAt (smoke-tests/test-staking-rewards.ts:222:51)
      at processTicksAndRejections (node:internal/process/task_queues:96:5)
      at async assertRewardsAtRoundBefore (smoke-tests/test-staking-rewards.ts:30:3)
      at async Context.<anonymous> (smoke-tests/test-staking-rewards.ts:20:7)

[4meta5]

Smoke test output now isn't as bad:

1) Verify staking rewards
       rewards are given as expected:

      AssertionError: Total rewarded events did not match total expected issuance for collators + delegators:
    255542519933313664430 != 491725050809954159919 

    Inflation was 236182530876640495489 less than expected for round 4440: expected '255542519933313664430' to equal '491725050809954159919'
      + expected - actual

      -255542519933313664430
      +491725050809954159919