aptos-core icon indicating copy to clipboard operation
aptos-core copied to clipboard

[gha] use beefy machine to replay-verify

Open msmouse opened this issue 2 years ago • 1 comments

Description

To make it faster.

Test Plan


This change is Reviewable

msmouse avatar Oct 21 '22 18:10 msmouse

4_Run ..githubactionsreplay-verify.txt:1145:2022-10-22T05:49:56.3000142Z [main process] spawning run_2_151500000_161000000
4_Run ..githubactionsreplay-verify.txt:1146:2022-10-22T05:49:56.3497294Z [main process] spawning run_3_161000000_170500000
4_Run ..githubactionsreplay-verify.txt:1147:2022-10-22T05:49:56.3983400Z [main process] spawning run_4_170500000_180000000
4_Run ..githubactionsreplay-verify.txt:1148:2022-10-22T05:49:56.4477032Z [main process] spawning run_5_180000000_189500000
4_Run ..githubactionsreplay-verify.txt:1149:2022-10-22T05:49:56.4973382Z [main process] spawning run_6_189500000_199000000
4_Run ..githubactionsreplay-verify.txt:1150:2022-10-22T05:49:56.5457797Z [main process] spawning run_7_199000000_208500000
4_Run ..githubactionsreplay-verify.txt:1151:2022-10-22T05:49:56.5955641Z [main process] spawning run_8_208500000_218000000
4_Run ..githubactionsreplay-verify.txt:1152:2022-10-22T05:49:56.6460554Z [main process] spawning run_9_218000000_227500000
4_Run ..githubactionsreplay-verify.txt:1153:2022-10-22T05:49:56.6955619Z [main process] spawning run_10_227500000_237000000
4_Run ..githubactionsreplay-verify.txt:1154:2022-10-22T05:49:56.7456902Z [main process] spawning run_11_237000000_246500000
4_Run ..githubactionsreplay-verify.txt:1155:2022-10-22T05:49:56.7966854Z [main process] spawning run_12_246500000_256000000
4_Run ..githubactionsreplay-verify.txt:1156:2022-10-22T05:49:56.8477828Z [main process] spawning run_13_256000000_265500000
4_Run ..githubactionsreplay-verify.txt:1157:2022-10-22T05:49:56.8977532Z [main process] spawning run_14_265500000_275000000
4_Run ..githubactionsreplay-verify.txt:1158:2022-10-22T05:49:56.9473882Z [main process] spawning run_15_275000000_284500000
4_Run ..githubactionsreplay-verify.txt:1159:2022-10-22T05:49:56.9970345Z [main process] spawning run_16_284500000_294000000
4_Run ..githubactionsreplay-verify.txt:26275:2022-10-22T07:42:12.2809874Z [main process] run 1 finished.
4_Run ..githubactionsreplay-verify.txt:30212:2022-10-22T07:59:50.3660795Z [main process] run 2 finished.
4_Run ..githubactionsreplay-verify.txt:31240:2022-10-22T08:04:27.0828614Z [main process] run 3 finished.
4_Run ..githubactionsreplay-verify.txt:34867:2022-10-22T08:21:30.2805184Z [main process] run 4 finished.
4_Run ..githubactionsreplay-verify.txt:35063:2022-10-22T08:22:26.7517463Z [main process] run 5 finished.
4_Run ..githubactionsreplay-verify.txt:35449:2022-10-22T08:24:14.5162010Z [main process] run 6 finished.
4_Run ..githubactionsreplay-verify.txt:39904:2022-10-22T08:29:10.0350480Z [main process] run 7 finished.
4_Run ..githubactionsreplay-verify.txt:40788:2022-10-22T08:34:06.7499131Z [main process] run 8 finished.
4_Run ..githubactionsreplay-verify.txt:41268:2022-10-22T08:37:22.0406835Z [main process] run 9 finished.
4_Run ..githubactionsreplay-verify.txt:41469:2022-10-22T08:38:04.6854485Z [main process] run 10 finished.
4_Run ..githubactionsreplay-verify.txt:43442:2022-10-22T08:53:43.3080525Z [main process] run 11 finished.
4_Run ..githubactionsreplay-verify.txt:43791:2022-10-22T08:57:08.1784250Z [main process] run 12 finished.
4_Run ..githubactionsreplay-verify.txt:47288:2022-10-22T09:59:55.7217463Z [main process] run 13 finished.
4_Run ..githubactionsreplay-verify.txt:47490:2022-10-22T10:04:24.2921244Z [main process] run 14 finished.
4_Run ..githubactionsreplay-verify.txt:47588:2022-10-22T10:08:01.8939629Z [main process] run 15 finished.
4_Run ..githubactionsreplay-verify.txt:47887:2022-10-22T10:42:01.5323353Z [main process] run 16 finished.
4_Run ..githubactionsreplay-verify.txt:47888:2022-10-22T10:42:01.5323975Z [main process] finished.

I feel we can save a lot of time by making more processes (the fast jobs finishes in 3 hours, and the slower ones 5 hours, because they are full of system txns). So let's try doubling the memory?

At the same time let's land this to get some signals ASAP?

msmouse avatar Oct 22 '22 15:10 msmouse

Forge is running suite land_blocking on 34b17143351b2ffbc687251eb530644c59689f18

github-actions[bot] avatar Oct 23 '22 06:10 github-actions[bot]

Forge is running suite compat on 2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 34b17143351b2ffbc687251eb530644c59689f18

github-actions[bot] avatar Oct 23 '22 06:10 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on 34b17143351b2ffbc687251eb530644c59689f18

performance benchmark with full nodes : 6975 TPS, 5699 ms latency, 8400 ms p99 latency,no expired txns
Test Ok

github-actions[bot] avatar Oct 23 '22 07:10 github-actions[bot]

:white_check_mark: Forge suite compat success on 2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 34b17143351b2ffbc687251eb530644c59689f18

Compatibility test results for 2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 34b17143351b2ffbc687251eb530644c59689f18 (PR)
1. Check liveness of validators at old version: 2d8b1b57553d869190f61df1aaf7f31a8fc19a7b
compatibility::simple-validator-upgrade::liveness-check : 7480 TPS, 5317 ms latency, 9000 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 34b17143351b2ffbc687251eb530644c59689f18
compatibility::simple-validator-upgrade::single-validator-upgrade : 4716 TPS, 8817 ms latency, 11800 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 34b17143351b2ffbc687251eb530644c59689f18
compatibility::simple-validator-upgrade::half-validator-upgrade : 4558 TPS, 8838 ms latency, 12300 ms p99 latency,no expired txns
4. upgrading second batch to new version: 34b17143351b2ffbc687251eb530644c59689f18
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7024 TPS, 5545 ms latency, 9500 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for 2d8b1b57553d869190f61df1aaf7f31a8fc19a7b ==> 34b17143351b2ffbc687251eb530644c59689f18 passed
Test Ok

github-actions[bot] avatar Oct 23 '22 07:10 github-actions[bot]