aptos-core [quorum store] better handling of proofs in proof coordinator

Description

The motivation is our metrics show failed quorums when there are late votes, even if the quorum was in fact reached before the late votes. There were subtle bugs in metrics when inline batches are used (before getting quorum) and/or batches with the same digests (either remotely or locally) are created.

Thinking a bit more, there's no reason to use digest in proof coordinator, so changing to use the batch info instead. This means we no longer have to disambiguate between batches that are possibly created with the same digests (either remotely or locally). This simplifies the code significantly while dealing with the above issues.

Type of Change

[ ] New feature
[x] Bug fix
[ ] Breaking change
[ ] Performance improvement
[ ] Refactoring
[ ] Dependency update
[ ] Documentation update
[ ] Tests

Which Components or Systems Does This Change Impact?

[x] Validator Node
[ ] Full Node (API, Indexer, etc.)
[ ] Move/Aptos Virtual Machine
[ ] Aptos Framework
[ ] Aptos CLI/SDK
[ ] Developer Infrastructure
[ ] Other (specify)

How Has This Been Tested?

Ran forge stable workload sweep test, which has gas cut

Key Areas to Review

Checklist

[ ] I have read and followed the CONTRIBUTING doc
[ ] I have performed a self-review of my own code
[ ] I have commented my code, particularly in hard-to-understand areas
[ ] I identified and added all stakeholders and component owners affected by this change as reviewers
[ ] I tested both happy and unhappy path of the functionality
[ ] I have made corresponding changes to the documentation

May 01 '24 21:05 bchocho

⏱️ 11h 40m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
forge-e2e-test / forge	2h 23m	🟩 🟩 🟩 🟩 🟩 (+3 more)
rust-images / rust-all	1h 52m	🟩 🟩 🟩 🟩 🟩 (+3 more)
rust-targeted-unit-tests	1h 37m	🟩 🟩 🟩 🟩 🟩 (+3 more)
rust-smoke-tests	1h 3m	🟩 🟩
execution-performance / single-node-performance	47m	🟩 🟩
rust-lints	44m	🟥 🟥 🟥 🟩 🟩 (+3 more)
run-tests-main-branch	33m	🟩 🟩 🟩 🟩 🟩 (+3 more)
rust-move-tests	29m	🟩 🟩 ⬜ 🟩 🟩 (+5 more)
forge-compat-test / forge	29m	🟩 🟩
framework-upgrade-determinator	27m	🟩 🟩 🟩 🟩 🟩 (+3 more)
cli-e2e-tests / run-cli-tests	16m	🟩 🟥
general-lints	15m	🟩 🟩 🟩 🟩 🟩 (+3 more)
check-dynamic-deps	13m	🟩 🟩 🟩 🟩 🟩 (+5 more)
check	8m	🟩 🟩
rust-build-cached-packages	8m	🟩 🟩
semgrep/ci	4m	🟩 🟩 🟩 🟩 🟩 (+5 more)
indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose	4m	🟩 🟩
node-api-compatibility-tests / node-api-compatibility-tests	2m	🟩 🟩
file_change_determinator	2m	🟩 🟩 ⬜ 🟩 🟩 (+5 more)
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+3 more)
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+3 more)
permission-check	41s	🟩 🟩 ⬜ 🟩 🟩 (+5 more)
permission-check	27s	🟩 🟩 🟩 🟩 🟩 (+5 more)
permission-check	26s	🟩 🟩 🟩 🟩 🟩 (+5 more)
permission-check	23s	🟩 🟩 ⬜ 🟩 🟩 (+5 more)
execution-performance / file_change_determinator	22s	🟩 🟩
determine-docker-build-metadata	22s	🟩 🟩 🟩 🟩 🟩 (+3 more)
permission-check	21s	🟩 🟩 🟩 🟩 🟩 (+3 more)

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
cli-e2e-tests / run-cli-tests	9m	6m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

May 01 '24 21:05 trunk-io[bot]

Forge is running suite `realistic_env_max_load` on `bf33cffe68986fdd83fde29a0561277b4696c6a8`

May 01 '24 21:05 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `bf33cffe68986fdd83fde29a0561277b4696c6a8`

two traffics test: inner traffic : committed: 8262 txn/s, latency: 4755 ms, (p50: 4500 ms, p90: 5600 ms, p99: 11100 ms), latency samples: 3561200
two traffics test : committed: 100 txn/s, latency: 1990 ms, (p50: 1900 ms, p90: 2300 ms, p99: 4300 ms), latency samples: 1780
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.207, avg: 0.204", "QsPosToProposal: max: 0.217, avg: 0.202", "ConsensusProposalToOrdered: max: 0.446, avg: 0.422", "ConsensusOrderedToCommit: max: 0.375, avg: 0.364", "ConsensusProposalToCommit: max: 0.792, avg: 0.786"]
Max round gap was 1 [limit 4] at version 922760. Max no progress secs was 5.061604 [limit 15] at version 3499113.
Test Ok

May 01 '24 21:05 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `e8b3a052a9c84518d377ebc2a11af07d3ffaee8d`

May 02 '24 21:05 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `e8b3a052a9c84518d377ebc2a11af07d3ffaee8d`

two traffics test: inner traffic : committed: 7306 txn/s, latency: 5339 ms, (p50: 5100 ms, p90: 6000 ms, p99: 14500 ms), latency samples: 3178440
two traffics test : committed: 100 txn/s, latency: 2008 ms, (p50: 1900 ms, p90: 2200 ms, p99: 5900 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.209, avg: 0.207", "QsPosToProposal: max: 0.394, avg: 0.255", "ConsensusProposalToOrdered: max: 0.471, avg: 0.445", "ConsensusOrderedToCommit: max: 0.411, avg: 0.397", "ConsensusProposalToCommit: max: 0.861, avg: 0.842"]
Max round gap was 1 [limit 4] at version 899523. Max no progress secs was 4.985288 [limit 15] at version 3184973.
Test Ok

May 02 '24 21:05 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `56951cc2df4d572be37da46d1f0efd96cf07f311`

May 02 '24 21:05 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `56951cc2df4d572be37da46d1f0efd96cf07f311`

two traffics test: inner traffic : committed: 7942 txn/s, latency: 4937 ms, (p50: 4700 ms, p90: 6000 ms, p99: 10800 ms), latency samples: 3431140
two traffics test : committed: 100 txn/s, latency: 2002 ms, (p50: 1900 ms, p90: 2200 ms, p99: 7700 ms), latency samples: 1660
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.211, avg: 0.205", "QsPosToProposal: max: 0.273, avg: 0.235", "ConsensusProposalToOrdered: max: 0.458, avg: 0.415", "ConsensusOrderedToCommit: max: 0.468, avg: 0.408", "ConsensusProposalToCommit: max: 0.897, avg: 0.822"]
Max round gap was 1 [limit 4] at version 1551820. Max no progress secs was 4.789089 [limit 15] at version 1551820.
Test Ok

May 02 '24 22:05 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `57ce6e9bf1cd107da14312908001d03c519477f2`

May 03 '24 18:05 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `57ce6e9bf1cd107da14312908001d03c519477f2`

two traffics test: inner traffic : committed: 7380 txn/s, submitted: 12989 txn/s, expired: 5609 txn/s, latency: 51931 ms, (p50: 58100 ms, p90: 59000 ms, p99: 60000 ms), latency samples: 79436
two traffics test : committed: 999 txn/s, latency: 1983 ms, (p50: 2100 ms, p90: 2400 ms, p99: 3600 ms), latency samples: 13740
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.182, avg: 0.179", "QsPosToProposal: max: 0.435, avg: 0.307", "ConsensusProposalToOrdered: max: 0.434, avg: 0.427", "ConsensusOrderedToCommit: max: 0.423, avg: 0.360", "ConsensusProposalToCommit: max: 0.854, avg: 0.787"]
Max round gap was 1 [limit 10] at version 379392. Max no progress secs was 5.535916 [limit 30] at version 2757873.
Test Ok

May 03 '24 18:05 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `9d261fb48e4048e8c438ebfd73a02434e0656d19`

May 03 '24 20:05 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `9d261fb48e4048e8c438ebfd73a02434e0656d19`

Workload                                 | submitted/s  | committed/s  | expired/s    | rejected/s   | chain txn/s  | latency      | p50 lat      | p90 lat      | p99 lat      | batch->pos   | pos->prop    | prop->order  | order->commit | actual dur  
0: CoinTransfer[20.0k]                   | 6008         | 6008         | 0            | 0            | 6008         | 3326         | 3300         | 3600         | 4000         | 0.218        | 0.214        | 0.459        | 0.338        | 54          
1: NoOp(100 modules)[20.0k]              | 5784         | 5784         | 0            | 0            | 5787         | 3423         | 3300         | 3800         | 4700         | 0.205        | 0.152        | 0.432        | 0.348        | 54          
2: ph0: CreateBurnerAccounts[20.0k]      | 6025         | 6025         | 0            | 0            | 6035         | 3323         | 3300         | 3600         | 3900         | 0.207        | 0.626        | 0.597        | 0.952        | 54          
2: ph1: ModifyGlobalResource[20.0k]      | 2441         | 2435         | 6            | 6            | 2438         | 7842         | 6000         | 14500        | 24100        | 0.207        | 0.625        | 0.597        | 0.639        | 54          
3: ph0: CreateBurnerAccounts[30.0k]      | 5996         | 5996         | 0            | 0            | 5967         | 5026         | 5000         | 5400         | 5700         | 0.215        | 0.170        | 0.451        | 0.413        | 54          
3: ph1: TokenV2AmbassadorMint[30.0k]     | 3424         | 3412         | 12           | 12           | 3767         | 8020         | 5500         | 17100        | 24300        | 0.214        | 0.160        | 0.451        | 0.407        | 58          
Test Ok

May 03 '24 20:05 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `d1ec4986f2d6df19749afb1ccfde813bd79de120`

May 03 '24 23:05 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `d1ec4986f2d6df19749afb1ccfde813bd79de120`

Workload                                 | submitted/s  | committed/s  | expired/s    | rejected/s   | chain txn/s  | latency      | p50 lat      | p90 lat      | p99 lat      | batch->pos   | pos->prop    | prop->order  | order->commit | actual dur  
0: CoinTransfer[20.0k]                   | 5997         | 5997         | 0            | 0            | 6002         | 3307         | 3200         | 3600         | 3900         | 0.205        | 0.136        | 0.396        | 0.341        | 181         
1: NoOp(100 modules)[20.0k]              | 5809         | 5809         | 0            | 0            | 5810         | 3412         | 3300         | 3700         | 5200         | 0.205        | 0.137        | 0.402        | 0.356        | 181         
2: ph0: CreateBurnerAccounts[20.0k]      | 5973         | 5973         | 0            | 0            | 5979         | 3325         | 3300         | 3600         | 4000         | 0.209        | 0.146        | 0.415        | 0.401        | 181         
2: ph1: ModifyGlobalResource[20.0k]      | 2845         | 2845         | 0            | 0            | 2840         | 6603         | 4600         | 12000        | 25000        | 0.207        | 0.178        | 0.474        | 0.839        | 181         
3: ph0: CreateBurnerAccounts[30.0k]      | 5953         | 5953         | 0            | 0            | 5942         | 5019         | 5000         | 5400         | 5700         | 0.214        | 0.147        | 0.429        | 0.429        | 181         
3: ph1: TokenV2AmbassadorMint[30.0k]     | 5099         | 5097         | 1            | 1            | 5106         | 5634         | 5100         | 6100         | 17900        | 0.223        | 0.169        | 0.447        | 0.544        | 181         
Test Ok

May 03 '24 23:05 github-actions[bot]

Forge is running suite `compat` on `01b24e7e3548382dd25440b39a0438a993387f12` ==> `a82f6b2aefe46bd89f386d8e044ff1647997a34e`

May 04 '24 00:05 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `a82f6b2aefe46bd89f386d8e044ff1647997a34e`

May 04 '24 00:05 github-actions[bot]

:white_check_mark: Forge suite `compat` success on `01b24e7e3548382dd25440b39a0438a993387f12` ==> `a82f6b2aefe46bd89f386d8e044ff1647997a34e`

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> a82f6b2aefe46bd89f386d8e044ff1647997a34e (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 4410 txn/s, latency: 7488 ms, (p50: 6500 ms, p90: 10400 ms, p99: 22900 ms), latency samples: 149940
2. Upgrading first Validator to new version: a82f6b2aefe46bd89f386d8e044ff1647997a34e
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1766 txn/s, latency: 16002 ms, (p50: 18700 ms, p90: 22500 ms, p99: 22900 ms), latency samples: 91860
3. Upgrading rest of first batch to new version: a82f6b2aefe46bd89f386d8e044ff1647997a34e
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1748 txn/s, latency: 16393 ms, (p50: 19000 ms, p90: 22500 ms, p99: 22900 ms), latency samples: 90900
4. upgrading second batch to new version: a82f6b2aefe46bd89f386d8e044ff1647997a34e
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3024 txn/s, latency: 10379 ms, (p50: 11400 ms, p90: 12600 ms, p99: 12900 ms), latency samples: 120960
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> a82f6b2aefe46bd89f386d8e044ff1647997a34e passed
Test Ok

May 04 '24 00:05 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `a82f6b2aefe46bd89f386d8e044ff1647997a34e`

two traffics test: inner traffic : committed: 8408 txn/s, latency: 4673 ms, (p50: 4500 ms, p90: 5400 ms, p99: 10200 ms), latency samples: 3624040
two traffics test : committed: 100 txn/s, latency: 1819 ms, (p50: 1800 ms, p90: 2000 ms, p99: 2300 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.206, avg: 0.202", "QsPosToProposal: max: 0.241, avg: 0.214", "ConsensusProposalToOrdered: max: 0.438, avg: 0.403", "ConsensusOrderedToCommit: max: 0.368, avg: 0.357", "ConsensusProposalToCommit: max: 0.772, avg: 0.760"]
Max round gap was 1 [limit 4] at version 1850495. Max no progress secs was 4.664958 [limit 15] at version 1850495.
Test Ok

May 04 '24 00:05 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `212a0b83229515a37ee8991e709249875e6ef6c1`

May 07 '24 00:05 github-actions[bot]

Forge is running suite `compat` on `01b24e7e3548382dd25440b39a0438a993387f12` ==> `212a0b83229515a37ee8991e709249875e6ef6c1`

May 07 '24 00:05 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `212a0b83229515a37ee8991e709249875e6ef6c1`

two traffics test: inner traffic : committed: 7290 txn/s, latency: 5379 ms, (p50: 5100 ms, p90: 6300 ms, p99: 11200 ms), latency samples: 3185900
two traffics test : committed: 100 txn/s, latency: 2050 ms, (p50: 1900 ms, p90: 2200 ms, p99: 5800 ms), latency samples: 1800
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.205, avg: 0.203", "QsPosToProposal: max: 0.355, avg: 0.248", "ConsensusProposalToOrdered: max: 0.517, avg: 0.455", "ConsensusOrderedToCommit: max: 0.413, avg: 0.392", "ConsensusProposalToCommit: max: 0.878, avg: 0.846"]
Max round gap was 1 [limit 4] at version 941226. Max no progress secs was 4.906599 [limit 15] at version 3201369.
Test Ok

May 07 '24 00:05 github-actions[bot]

:white_check_mark: Forge suite `compat` success on `01b24e7e3548382dd25440b39a0438a993387f12` ==> `212a0b83229515a37ee8991e709249875e6ef6c1`

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 212a0b83229515a37ee8991e709249875e6ef6c1 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 6724 txn/s, latency: 4919 ms, (p50: 4800 ms, p90: 8100 ms, p99: 9600 ms), latency samples: 235360
2. Upgrading first Validator to new version: 212a0b83229515a37ee8991e709249875e6ef6c1
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1830 txn/s, latency: 15710 ms, (p50: 18400 ms, p90: 22200 ms, p99: 22600 ms), latency samples: 91500
3. Upgrading rest of first batch to new version: 212a0b83229515a37ee8991e709249875e6ef6c1
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1497 txn/s, latency: 17218 ms, (p50: 19200 ms, p90: 23400 ms, p99: 25700 ms), latency samples: 85340
4. upgrading second batch to new version: 212a0b83229515a37ee8991e709249875e6ef6c1
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3621 txn/s, latency: 8807 ms, (p50: 9600 ms, p90: 12600 ms, p99: 12700 ms), latency samples: 144840
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 212a0b83229515a37ee8991e709249875e6ef6c1 passed
Test Ok

May 07 '24 00:05 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `767ca352546df746afe018cde9792ed2f544a082`

May 16 '24 03:05 github-actions[bot]

Forge is running suite `compat` on `01b24e7e3548382dd25440b39a0438a993387f12` ==> `767ca352546df746afe018cde9792ed2f544a082`

May 16 '24 03:05 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `767ca352546df746afe018cde9792ed2f544a082`

two traffics test: inner traffic : committed: 7364.309181594969 txn/s, latency: 5322.784293173978 ms, (p50: 5200 ms, p90: 6300 ms, p99: 10600 ms), latency samples: 3182820
two traffics test : committed: 100.06756114277631 txn/s, latency: 1965.3131868131868 ms, (p50: 1900 ms, p90: 2200 ms, p99: 5700 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.220, avg: 0.205", "QsPosToProposal: max: 0.273, avg: 0.262", "ConsensusProposalToOrdered: max: 0.465, avg: 0.425", "ConsensusOrderedToCommit: max: 0.439, avg: 0.424", "ConsensusProposalToCommit: max: 0.861, avg: 0.850"]
Max round gap was 1 [limit 4] at version 1655347. Max no progress secs was 4.539366 [limit 15] at version 1655347.
Test Ok

May 16 '24 03:05 github-actions[bot]