aptos-core icon indicating copy to clipboard operation
aptos-core copied to clipboard

Published 20 hours ago • verified publisher icon aptos-labs

[Block Executor] Parallelize resource group accesses

Open gelash opened this issue 9 months ago • 1 comments

Ready for review but cleaning up tests and adding tests for metadata.

Resource group validation must use entries marked as estimates instead of failing, and on the flip side, validation should occur after execution is finished not only when writing outside of previous write set (vanilla Block-STM rule), but when these (size or metadata) are changed after previous incarnation.

Type of Change

  • [x] New feature
  • [ ] Bug fix
  • [ ] Breaking change
  • [ ] Performance improvement
  • [ ] Refactoring
  • [ ] Dependency update
  • [ ] Documentation update
  • [ ] Tests

Which Components or Systems Does This Change Impact?

  • [x] Validator Node
  • [ ] Full Node (API, Indexer, etc.)
  • [ ] Move/Aptos Virtual Machine
  • [ ] Aptos Framework
  • [ ] Aptos CLI/SDK
  • [ ] Developer Infrastructure
  • [ ] Other (specify)

How Has This Been Tested?

Passes resource group tests and did not pass without all proper changes, i.e. confirming prop-tests used to generate problematic corner cases for the size TODO: adding tests for metadata

Key Areas to Review

Checklist

  • [x] I have read and followed the CONTRIBUTING doc
  • [ ] I have performed a self-review of my own code
  • [ ] I have commented my code, particularly in hard-to-understand areas
  • [ ] I identified and added all stakeholders and component owners affected by this change as reviewers
  • [ ] I tested both happy and unhappy path of the functionality
  • [ ] I have made corresponding changes to the documentation

gelash avatar Apr 30 '24 16:04 gelash

⏱️ 2h 44m total CI duration on this PR
Job Cumulative Duration Recent Runs
execution-performance / single-node-performance 1h 10m 🟥 🟥 
rust-move-tests 38m ⬜ 🟩 🟩 
rust-targeted-unit-tests 13m 🟥 🟥 
rust-images / rust-all 12m 🟩 
run-tests-main-branch 9m 🟩 🟩 
rust-lints 7m 🟥 🟥 
check-dynamic-deps 4m 🟩 🟩 🟩 
general-lints 4m 🟩 🟩 
framework-upgrade-determinator 3m 🟥 
indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose 2m 🟩 
semgrep/ci 2m 🟩 🟩 🟩 
execution-performance / file_change_determinator 22s 🟩 🟩 
file_change_determinator 22s 🟩 🟩 
file_change_determinator 21s 🟩 🟩 
permission-check 11s 🟩 🟩 🟩 
permission-check 10s 🟩 🟩 🟩 
file_change_determinator 9s 🟩 
permission-check 6s 🟩 ⬜ 🟩 
permission-check 6s 🟩 🟩 
permission-check 3s 🟩 
determine-docker-build-metadata 2s 🟩 

🚨 1 job on the last run was significantly faster/slower than expected

Job Duration vs 7d avg Delta
execution-performance / single-node-performance 34m 25m +39%

settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io

trunk-io[bot] avatar Apr 30 '24 16:04 trunk-io[bot]

Was wondering when those metadata can be changed? IIUC they should stay immutable? Am I missing anything?

runtian-zhou avatar May 31 '24 07:05 runtian-zhou

Was wondering when those metadata can be changed? IIUC they should stay immutable? Am I missing anything?

@runtian-zhou : speculation, i.e. it can change from one execution to another. We don't wait or fail validations on the estimates on metadata anymore - as that kills resource group parallelism, but instead read it optimistically even if it is an estimate, but on the flip side now we should re-validate the suffix if re-execution changes what was there (and maybe read) before.

gelash avatar May 31 '24 15:05 gelash

Forge is running suite framework_upgrade on a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

github-actions[bot] avatar Jun 07 '24 13:06 github-actions[bot]

Forge is running suite realistic_env_max_load on 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

github-actions[bot] avatar Jun 07 '24 13:06 github-actions[bot]

Forge is running suite compat on a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

github-actions[bot] avatar Jun 07 '24 13:06 github-actions[bot]

:white_check_mark: Forge suite compat success on a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

Compatibility test results for a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5 (PR)
1. Check liveness of validators at old version: a68e71c05caebf01504d4499110f3fba213fb53d
compatibility::simple-validator-upgrade::liveness-check : committed: 8943.476034830937 txn/s, latency: 3251.610468195391 ms, (p50: 2700 ms, p90: 5400 ms, p99: 12900 ms), latency samples: 354980
2. Upgrading first Validator to new version: 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6925.879868774864 txn/s, latency: 4348.2069375414185 ms, (p50: 4300 ms, p90: 6100 ms, p99: 6500 ms), latency samples: 241440
3. Upgrading rest of first batch to new version: 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5928.908778878722 txn/s, latency: 5218.943979582101 ms, (p50: 5300 ms, p90: 8000 ms, p99: 10400 ms), latency samples: 209620
4. upgrading second batch to new version: 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11547.134188552864 txn/s, latency: 2728.883851229116 ms, (p50: 2600 ms, p90: 3800 ms, p99: 5200 ms), latency samples: 375880
5. check swarm health
Compatibility test for a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5 passed
Test Ok

github-actions[bot] avatar Jun 07 '24 13:06 github-actions[bot]

:white_check_mark: Forge suite realistic_env_max_load success on 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

two traffics test: inner traffic : committed: 9319.105445450577 txn/s, latency: 4218.422229435291 ms, (p50: 4200 ms, p90: 4500 ms, p99: 5400 ms), latency samples: 4020480
two traffics test : committed: 100.06723704044977 txn/s, latency: 2030.2412790697674 ms, (p50: 2000 ms, p90: 2200 ms, p99: 4500 ms), latency samples: 1720
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.233, avg: 0.214", "QsPosToProposal: max: 2.017, avg: 1.932", "ConsensusProposalToOrdered: max: 0.320, avg: 0.286", "ConsensusOrderedToCommit: max: 0.368, avg: 0.356", "ConsensusProposalToCommit: max: 0.655, avg: 0.642"]
Max round gap was 1 [limit 4] at version 1929332. Max no progress secs was 5.186178 [limit 15] at version 1929332.
Test Ok

github-actions[bot] avatar Jun 07 '24 13:06 github-actions[bot]

:white_check_mark: Forge suite framework_upgrade success on a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

Compatibility test results for a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5 (PR)
Upgrade the nodes to version: 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1194.8517494088626 txn/s, submitted: 1196.6304643316892 txn/s, failed submission: 1.7787149228267398 txn/s, expired: 1.7787149228267398 txn/s, latency: 2685.438956084853 ms, (p50: 2000 ms, p90: 5100 ms, p99: 8400 ms), latency samples: 107480
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1164.7495861368234 txn/s, submitted: 1165.9520944850958 txn/s, failed submission: 1.2025083482725825 txn/s, expired: 1.2025083482725825 txn/s, latency: 2759.622073095189 ms, (p50: 2100 ms, p90: 5200 ms, p99: 9900 ms), latency samples: 96860
5. check swarm health
Compatibility test for a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5 passed
Upgrade the remaining nodes to version: 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1113.469112158987 txn/s, submitted: 1115.8858762145805 txn/s, failed submission: 2.416764055593697 txn/s, expired: 2.416764055593697 txn/s, latency: 2714.497829518548 ms, (p50: 2100 ms, p90: 4500 ms, p99: 10500 ms), latency samples: 101360
Test Ok

github-actions[bot] avatar Jun 07 '24 13:06 github-actions[bot]

Forge is running suite realistic_env_max_load on 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

github-actions[bot] avatar Jun 07 '24 20:06 github-actions[bot]

Forge is running suite framework_upgrade on a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

github-actions[bot] avatar Jun 07 '24 20:06 github-actions[bot]

Forge is running suite compat on a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

github-actions[bot] avatar Jun 07 '24 20:06 github-actions[bot]

:white_check_mark: Forge suite compat success on a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

Compatibility test results for a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5 (PR)
1. Check liveness of validators at old version: a68e71c05caebf01504d4499110f3fba213fb53d
compatibility::simple-validator-upgrade::liveness-check : committed: 9421.105960467117 txn/s, latency: 3525.40857714723 ms, (p50: 2600 ms, p90: 6300 ms, p99: 27900 ms), latency samples: 339740
2. Upgrading first Validator to new version: 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6164.5444005799445 txn/s, latency: 5178.910115553926 ms, (p50: 4600 ms, p90: 6700 ms, p99: 7200 ms), latency samples: 218080
3. Upgrading rest of first batch to new version: 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 5944.403423273393 txn/s, latency: 5379.761410610535 ms, (p50: 4500 ms, p90: 7400 ms, p99: 7900 ms), latency samples: 209980
4. upgrading second batch to new version: 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8942.130630177384 txn/s, latency: 3408.2302331162155 ms, (p50: 2800 ms, p90: 5200 ms, p99: 7400 ms), latency samples: 291700
5. check swarm health
Compatibility test for a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5 passed
Test Ok

github-actions[bot] avatar Jun 07 '24 20:06 github-actions[bot]

:x: Forge suite realistic_env_max_load failure on 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

two traffics test: inner traffic : committed: 9429.631519995724 txn/s, latency: 4169.809671350649 ms, (p50: 4100 ms, p90: 4500 ms, p99: 12300 ms), latency samples: 4066340
two traffics test : committed: 100.05413863854707 txn/s, latency: 2119.9113636363636 ms, (p50: 2000 ms, p90: 2200 ms, p99: 10300 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.231, avg: 0.212", "QsPosToProposal: max: 1.924, avg: 1.866", "ConsensusProposalToOrdered: max: 0.309, avg: 0.283", "ConsensusOrderedToCommit: max: 0.364, avg: 0.354", "ConsensusProposalToCommit: max: 0.649, avg: 0.637"]
Test Failed: check for success

Caused by:
    error!() count in validator logs was 1, and must be 0

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_no_errors::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:571:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_for_success::{{closure}}
             at ./testsuite/forge/src/success_criteria.rs:328:42
   3: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   4: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
   5: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
   6: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
   7: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
   8: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
   9: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  10: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  11: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  12: aptos_forge::interface::network::NetworkContext::check_for_success
             at ./testsuite/forge/src/interface/network.rs:71:9
  13: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:229:13
  14: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:499:9
  15: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  16: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  17: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  18: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  19: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  20: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  21: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  22: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  23: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  24: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  25: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  26: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  27: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  28: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  29: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-13133-1717790703-8cd7ad5d650c7447cd5ecbf777f530e3f","timestamp":"2024-06-07T20:16:33.197841Z","message":"Deleting namespace forge-e2e-pr-13133: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-13133-1717790703-8cd7ad5d650c7447cd5ecbf777f530e3f","timestamp":"2024-06-07T20:16:33.197873Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-13133"}
Failed to run tests:
Tests Failed

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge188-0       1/1     Running     0          10m
aptos-node-0-validator-0                1/1     Running     0          10m
aptos-node-1-fullnode-eforge188-0       1/1     Running     0          10m
aptos-node-1-validator-0                1/1     Running     0          10m
aptos-node-2-fullnode-eforge188-0       1/1     Running     0          10m
aptos-node-2-validator-0                1/1     Running     0          10m
aptos-node-3-fullnode-eforge188-0       1/1     Running     0          10m
aptos-node-3-validator-0                1/1     Running     0          10m
aptos-node-4-fullnode-eforge188-0       1/1     Running     0          10m
aptos-node-4-validator-0                1/1     Running     0          10m
aptos-node-5-validator-0                1/1     Running     0          10m
aptos-node-6-validator-0                1/1     Running     0          10m
genesis-aptos-genesis-eforge188-6swlg   0/1     Completed   0          11m

github-actions[bot] avatar Jun 07 '24 20:06 github-actions[bot]

:white_check_mark: Forge suite framework_upgrade success on a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

Compatibility test results for a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5 (PR)
Upgrade the nodes to version: 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1269.5853007365724 txn/s, submitted: 1271.9645835340548 txn/s, failed submission: 2.379282797482332 txn/s, expired: 2.379282797482332 txn/s, latency: 2513.012218890555 ms, (p50: 1800 ms, p90: 4500 ms, p99: 9800 ms), latency samples: 106720
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1060.0056549549374 txn/s, submitted: 1062.4992525179196 txn/s, failed submission: 2.4935975629821026 txn/s, expired: 2.4935975629821026 txn/s, latency: 2873.70124037639 ms, (p50: 2100 ms, p90: 4900 ms, p99: 11400 ms), latency samples: 93520
5. check swarm health
Compatibility test for a68e71c05caebf01504d4499110f3fba213fb53d ==> 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5 passed
Upgrade the remaining nodes to version: 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1106.7182904193862 txn/s, submitted: 1109.8202035456868 txn/s, failed submission: 3.101913126300582 txn/s, expired: 3.101913126300582 txn/s, latency: 2853.3794294294294 ms, (p50: 2100 ms, p90: 5300 ms, p99: 9600 ms), latency samples: 99900
Test Ok

github-actions[bot] avatar Jun 07 '24 20:06 github-actions[bot]

Forge is running suite realistic_env_max_load on 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

github-actions[bot] avatar Jun 07 '24 21:06 github-actions[bot]

:white_check_mark: Forge suite realistic_env_max_load success on 8cd7ad5d650c7447cd5ecbf777f530e3f564e2a5

two traffics test: inner traffic : committed: 9338.541419582534 txn/s, latency: 4212.131463449755 ms, (p50: 4200 ms, p90: 4500 ms, p99: 5400 ms), latency samples: 4027880
two traffics test : committed: 99.92221079808361 txn/s, latency: 2064.6770588235295 ms, (p50: 2000 ms, p90: 2300 ms, p99: 4700 ms), latency samples: 1700
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.236, avg: 0.216", "QsPosToProposal: max: 1.989, avg: 1.938", "ConsensusProposalToOrdered: max: 0.298, avg: 0.285", "ConsensusOrderedToCommit: max: 0.384, avg: 0.371", "ConsensusProposalToCommit: max: 0.669, avg: 0.656"]
Max round gap was 1 [limit 4] at version 1972415. Max no progress secs was 5.28927 [limit 15] at version 1972415.
Test Ok

github-actions[bot] avatar Jun 07 '24 21:06 github-actions[bot]