aptos-core icon indicating copy to clipboard operation
aptos-core copied to clipboard
verified publisher icon aptos-labs

Resend "[pipeline] pre commit (#13604)" (#14372)

Open msmouse opened this issue 1 year ago • 19 comments

reverted in #14372

fix epoch ending logic in new commit

see inline comments

Description

Type of Change

  • [x] New feature

Which Components or Systems Does This Change Impact?

  • [x] Validator Node

How Has This Been Tested?

tests

msmouse avatar Aug 22 '24 00:08 msmouse

⏱️ 28h 52m total CI duration on this PR
Job Cumulative Duration Recent Runs
execution-performance / single-node-performance 8h 46m 🟥 🟩 🟩 🟩 🟩 (+21 more)
forge-compat-test / forge 3h 16m 🟥 🟩 🟩 ⬜ 🟩 (+10 more)
forge-e2e-test / forge 3h 12m 🟩 🟩 ⬜ 🟥 ⬜ (+11 more)
execution-performance / test-target-determinator 2h 28m 🟩 🟩 🟩 🟩 🟩 (+20 more)
test-target-determinator 1h 53m ⬜ ⬜ ⬜ 🟩 🟩 (+23 more)
check 1h 11m ⬜ ⬜ ⬜ 🟩 🟩 (+20 more)
indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose 1h 4m 🟩 🟩 ⬜ 🟥 🟩 (+11 more)
rust-cargo-deny 53m 🟩 ⬜ ⬜ 🟩 🟩 (+20 more)
general-lints 40m 🟩 ⬜ ⬜ 🟩 🟩 (+20 more)
check-dynamic-deps 35m 🟩 🟩 🟩 🟩 🟩 (+22 more)
forge-framework-upgrade-test / forge 28m 🟥 🟩 ⬜ 
semgrep/ci 11m 🟩 🟩 🟩 🟩 🟩 (+23 more)
rust-move-tests 9m 🟩 
rust-move-tests 9m 🟩 
rust-move-tests 9m 🟩 
rust-move-tests 9m 🟩 
rust-move-tests 9m 🟩 
rust-move-tests 9m 🟩 
rust-move-tests 9m 🟩 
rust-move-tests 9m 🟩 
rust-move-tests 8m 🟩 
rust-move-tests 8m 🟩 
rust-move-tests 8m 🟩 
rust-move-tests 8m 🟩 
rust-move-tests 8m 🟩 
rust-move-tests 8m 🟩 
rust-move-tests 8m 🟩 
file_change_determinator 6m 🟩 🟩 🟩 🟩 🟩 (+25 more)
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
file_change_determinator 5m 🟩 🟩 🟩 🟩 🟩 (+22 more)
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
rust-doc-tests 5m 🟩 
file_change_determinator 5m ⬜ 🟩 🟩 🟩 🟩 (+23 more)
rust-doc-tests 4m 🟩 
rust-doc-tests 4m 🟩 
rust-doc-tests 4m ⬜ 
rust-move-tests 4m ⬜ 
rust-move-tests 4m 🟥 
rust-doc-tests 2m ⬜ 
rust-move-tests 2m ⬜ 
rust-move-tests 2m ⬜ 
rust-doc-tests 2m ⬜ 
rust-move-tests 2m ⬜ 
permission-check 2m 🟩 🟩 🟩 🟩 🟩 (+25 more)
rust-doc-tests 2m ⬜ 
permission-check 2m 🟩 🟩 🟩 🟩 🟩 (+25 more)
determine-docker-build-metadata 1m 🟩 🟩 🟩 🟩 🟩 (+23 more)
permission-check 1m 🟩 🟩 🟩 🟩 🟩 (+25 more)
permission-check 1m 🟩 🟩 🟩 🟩 🟩 (+25 more)
permission-check 1m 🟩 🟩 🟩 🟩 🟩 (+23 more)
rust-move-tests 1m ⬜ 
rust-doc-tests 1m ⬜ 
rust-move-tests 1m ⬜ 
rust-move-tests 1m ⬜ 
rust-doc-tests 1m ⬜ 
rust-move-tests 58s ⬜ 
rust-doc-tests 58s ⬜ 
Backport PR 3s 🟥 
rust-move-tests 3s ⬜ 
permission-check 2s 🟩 
rust-doc-tests 1s ⬜ 
rust-move-tests 1s ⬜ 
rust-doc-tests 1s ⬜ 
rust-doc-tests 1s ⬜ 
rust-move-tests 1s ⬜ 

settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io

trunk-io[bot] avatar Aug 22 '24 00:08 trunk-io[bot]

Forge is running suite realistic_env_max_load on 01bf1ad0d0840f5f5b1238ab45cb28a64050d96d

github-actions[bot] avatar Aug 22 '24 00:08 github-actions[bot]

Forge is running suite compat on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 01bf1ad0d0840f5f5b1238ab45cb28a64050d96d

github-actions[bot] avatar Aug 22 '24 00:08 github-actions[bot]

:white_check_mark: Forge suite realistic_env_max_load success on 01bf1ad0d0840f5f5b1238ab45cb28a64050d96d

two traffics test: inner traffic : committed: 13403.67 txn/s, latency: 2972.49 ms, (p50: 2800 ms, p90: 3300 ms, p99: 3900 ms), latency samples: 5096360
two traffics test : committed: 100.03 txn/s, latency: 2727.14 ms, (p50: 2600 ms, p90: 3100 ms, p99: 10700 ms), latency samples: 1740
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.244, avg: 0.216", "QsPosToProposal: max: 0.236, avg: 0.197", "ConsensusProposalToOrdered: max: 0.322, avg: 0.292", "ConsensusOrderedToCommit: max: 0.453, avg: 0.436", "ConsensusProposalToCommit: max: 0.744, avg: 0.729"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.94s no progress at version 2724575 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.08s no progress at version 2724573 (avg 8.08s) [limit 15].
Test Ok

github-actions[bot] avatar Aug 22 '24 00:08 github-actions[bot]

:white_check_mark: Forge suite compat success on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 01bf1ad0d0840f5f5b1238ab45cb28a64050d96d

Compatibility test results for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 01bf1ad0d0840f5f5b1238ab45cb28a64050d96d (PR)
1. Check liveness of validators at old version: d1bf834728a0cf166d993f4728dfca54f3086fb0
compatibility::simple-validator-upgrade::liveness-check : committed: 9807.00 txn/s, latency: 3416.67 ms, (p50: 2100 ms, p90: 8700 ms, p99: 27500 ms), latency samples: 380800
2. Upgrading first Validator to new version: 01bf1ad0d0840f5f5b1238ab45cb28a64050d96d
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6920.19 txn/s, latency: 3947.89 ms, (p50: 4400 ms, p90: 5500 ms, p99: 5600 ms), latency samples: 123560
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7304.71 txn/s, latency: 4343.67 ms, (p50: 4400 ms, p90: 6700 ms, p99: 6900 ms), latency samples: 241660
3. Upgrading rest of first batch to new version: 01bf1ad0d0840f5f5b1238ab45cb28a64050d96d
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7423.70 txn/s, latency: 3862.07 ms, (p50: 4300 ms, p90: 4700 ms, p99: 4700 ms), latency samples: 140940
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6324.30 txn/s, latency: 4648.21 ms, (p50: 4700 ms, p90: 5200 ms, p99: 6600 ms), latency samples: 240760
4. upgrading second batch to new version: 01bf1ad0d0840f5f5b1238ab45cb28a64050d96d
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 1513.01 txn/s, latency: 15189.08 ms, (p50: 14600 ms, p90: 27700 ms, p99: 29200 ms), latency samples: 65160
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 8638.00 txn/s, latency: 3749.65 ms, (p50: 3000 ms, p90: 8300 ms, p99: 8900 ms), latency samples: 288080
5. check swarm health
Compatibility test for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 01bf1ad0d0840f5f5b1238ab45cb28a64050d96d passed
Test Ok

github-actions[bot] avatar Aug 22 '24 00:08 github-actions[bot]

Forge is running suite realistic_env_max_load on 6917d7138a82ec809bdb2345a8192b9bc8b5d206

github-actions[bot] avatar Aug 22 '24 04:08 github-actions[bot]

Forge is running suite compat on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 6917d7138a82ec809bdb2345a8192b9bc8b5d206

github-actions[bot] avatar Aug 22 '24 04:08 github-actions[bot]

:white_check_mark: Forge suite realistic_env_max_load success on 6917d7138a82ec809bdb2345a8192b9bc8b5d206

two traffics test: inner traffic : committed: 13296.90 txn/s, latency: 2990.38 ms, (p50: 2700 ms, p90: 3300 ms, p99: 6600 ms), latency samples: 5055760
two traffics test : committed: 99.94 txn/s, latency: 2718.79 ms, (p50: 2400 ms, p90: 3100 ms, p99: 6900 ms), latency samples: 1740
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.296, avg: 0.230", "QsPosToProposal: max: 0.295, avg: 0.232", "ConsensusProposalToOrdered: max: 0.339, avg: 0.302", "ConsensusOrderedToCommit: max: 0.538, avg: 0.474", "ConsensusProposalToCommit: max: 0.837, avg: 0.776"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.84s no progress at version 12571 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 7.80s no progress at version 2749516 (avg 7.80s) [limit 15].
Test Ok

github-actions[bot] avatar Aug 22 '24 04:08 github-actions[bot]

:white_check_mark: Forge suite compat success on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 6917d7138a82ec809bdb2345a8192b9bc8b5d206

Compatibility test results for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 6917d7138a82ec809bdb2345a8192b9bc8b5d206 (PR)
1. Check liveness of validators at old version: d1bf834728a0cf166d993f4728dfca54f3086fb0
compatibility::simple-validator-upgrade::liveness-check : committed: 9634.71 txn/s, latency: 3661.08 ms, (p50: 2100 ms, p90: 4600 ms, p99: 24200 ms), latency samples: 448420
2. Upgrading first Validator to new version: 6917d7138a82ec809bdb2345a8192b9bc8b5d206
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7023.25 txn/s, latency: 3722.58 ms, (p50: 4100 ms, p90: 4500 ms, p99: 4600 ms), latency samples: 144220
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7341.12 txn/s, latency: 4303.41 ms, (p50: 4400 ms, p90: 6600 ms, p99: 7000 ms), latency samples: 241320
3. Upgrading rest of first batch to new version: 6917d7138a82ec809bdb2345a8192b9bc8b5d206
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6749.62 txn/s, latency: 4041.80 ms, (p50: 4300 ms, p90: 5600 ms, p99: 5700 ms), latency samples: 120540
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7186.63 txn/s, latency: 4477.20 ms, (p50: 4600 ms, p90: 6400 ms, p99: 6800 ms), latency samples: 239720
4. upgrading second batch to new version: 6917d7138a82ec809bdb2345a8192b9bc8b5d206
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 6886.14 txn/s, latency: 3979.99 ms, (p50: 2500 ms, p90: 7300 ms, p99: 8200 ms), latency samples: 120180
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10658.91 txn/s, latency: 2806.14 ms, (p50: 2500 ms, p90: 2900 ms, p99: 8000 ms), latency samples: 405680
5. check swarm health
Compatibility test for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 6917d7138a82ec809bdb2345a8192b9bc8b5d206 passed
Test Ok

github-actions[bot] avatar Aug 22 '24 04:08 github-actions[bot]

Forge is running suite realistic_env_max_load on 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf

github-actions[bot] avatar Aug 22 '24 17:08 github-actions[bot]

Forge is running suite compat on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf

github-actions[bot] avatar Aug 22 '24 17:08 github-actions[bot]

:white_check_mark: Forge suite compat success on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf

Compatibility test results for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf (PR)
1. Check liveness of validators at old version: d1bf834728a0cf166d993f4728dfca54f3086fb0
compatibility::simple-validator-upgrade::liveness-check : committed: 12773.15 txn/s, latency: 2595.90 ms, (p50: 2200 ms, p90: 4400 ms, p99: 7800 ms), latency samples: 427600
2. Upgrading first Validator to new version: 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7511.27 txn/s, latency: 3804.63 ms, (p50: 4300 ms, p90: 4800 ms, p99: 4900 ms), latency samples: 138180
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7539.85 txn/s, latency: 4232.73 ms, (p50: 4600 ms, p90: 5700 ms, p99: 6300 ms), latency samples: 250020
3. Upgrading rest of first batch to new version: 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7574.85 txn/s, latency: 3792.72 ms, (p50: 4300 ms, p90: 4700 ms, p99: 4800 ms), latency samples: 142760
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7528.74 txn/s, latency: 4304.88 ms, (p50: 4600 ms, p90: 5100 ms, p99: 5400 ms), latency samples: 248300
4. upgrading second batch to new version: 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11161.58 txn/s, latency: 2487.41 ms, (p50: 2400 ms, p90: 3300 ms, p99: 4400 ms), latency samples: 194600
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11804.77 txn/s, latency: 2639.79 ms, (p50: 2600 ms, p90: 3100 ms, p99: 3700 ms), latency samples: 385360
5. check swarm health
Compatibility test for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf passed
Test Ok

github-actions[bot] avatar Aug 22 '24 17:08 github-actions[bot]

:white_check_mark: Forge suite realistic_env_max_load success on 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf

two traffics test: inner traffic : committed: 13129.32 txn/s, latency: 3058.41 ms, (p50: 2900 ms, p90: 3300 ms, p99: 12100 ms), latency samples: 4992040
two traffics test : committed: 99.89 txn/s, latency: 2781.61 ms, (p50: 2500 ms, p90: 3000 ms, p99: 10500 ms), latency samples: 1720
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.288, avg: 0.232", "QsPosToProposal: max: 0.366, avg: 0.263", "ConsensusProposalToOrdered: max: 0.347, avg: 0.306", "ConsensusOrderedToCommit: max: 0.601, avg: 0.538", "ConsensusProposalToCommit: max: 0.906, avg: 0.844"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.88s no progress at version 5358649 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.17s no progress at version 5358647 (avg 8.16s) [limit 15].
Test Ok

github-actions[bot] avatar Aug 22 '24 17:08 github-actions[bot]

Forge is running suite realistic_env_max_load on 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf

github-actions[bot] avatar Aug 22 '24 20:08 github-actions[bot]

Forge is running suite compat on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf

github-actions[bot] avatar Aug 22 '24 20:08 github-actions[bot]

Forge is running suite framework_upgrade on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf

github-actions[bot] avatar Aug 22 '24 20:08 github-actions[bot]

:x: Forge suite framework_upgrade failure on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf

Compatibility test results for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf (PR)
Upgrade the nodes to version: 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf
Test Failed: Tried executing 10 txns, request counters: "success 0, failed submit [0], failed wait [10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10], by client: [(0, 0, 60): http://aptos-node-0-validator.forge-framework-upgrade-pr-14378.svc:8080/v1/]\n[(0, 0, 40): http://aptos-node-3-validator.forge-framework-upgrade-pr-14378.svc:8080/v1/]\n[(0, 0, 50): http://aptos-node-1-validator.forge-framework-upgrade-pr-14378.svc:8080/v1/]\n[(0, 0, 30): http://aptos-node-2-validator.forge-framework-upgrade-pr-14378.svc:8080/v1/]"

Caused by:
    Unknown error Ledger on endpoint (http://aptos-node-2-validator.forge-framework-upgrade-pr-14378.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (207c7661) might still succeed.

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/result.rs:1964:27
   2: aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:140:28
   3: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/maybe_done.rs:95:38
   4: <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/join_all.rs:143:24
   5: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:322:10
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   7: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_seed_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:404:18
   8: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:304:10
   9: aptos_transaction_emitter_lib::emitter::account_minter::bulk_create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:689:14
  10: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:744:10
  11: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:858:14
  12: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:890:14
  13: aptos_testcases::generate_traffic::{{closure}}
             at ./testsuite/testcases/src/lib.rs:177:10
  14: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/framework_upgrade.rs:86:73
  15: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  16: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  17: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  18: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  19: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  20: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  21: tokio::runtime::handle::Handle::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:310:22
  22: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  23: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:309:9
  24: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:611:49
  25: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:684:11
  26: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:611:30
  27: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
  28: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
  29: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  30: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  31: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  32: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  33: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  34: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  35: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  36: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  37: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  38: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  39: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  40: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  41: main
  42: __libc_start_main
  43: _start
Trailing Log Lines:
  38: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  39: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  40: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  41: main
  42: __libc_start_main
  43: _start


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-14378-1724360368-d1bf834728a0cf166d9","timestamp":"2024-08-22T21:08:14.387255Z","message":"Deleting namespace forge-framework-upgrade-pr-14378: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-14378-1724360368-d1bf834728a0cf166d9","timestamp":"2024-08-22T21:08:14.387278Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-14378"}
Failed to run tests:
Tests Failed

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:636:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                1/1     Running     0          6m15s
aptos-node-1-validator-0                1/1     Running     0          4m44s
aptos-node-2-validator-0                1/1     Running     0          7m55s
aptos-node-3-validator-0                1/1     Running     0          7m55s
genesis-aptos-genesis-eforge214-bgg2g   0/1     Completed   0          8m33s

github-actions[bot] avatar Aug 22 '24 21:08 github-actions[bot]

:white_check_mark: Forge suite realistic_env_max_load success on 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf

two traffics test: inner traffic : committed: 12588.06 txn/s, latency: 3161.43 ms, (p50: 3000 ms, p90: 3600 ms, p99: 4200 ms), latency samples: 4786220
two traffics test : committed: 99.96 txn/s, latency: 2562.11 ms, (p50: 2400 ms, p90: 2900 ms, p99: 8200 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.273, avg: 0.226", "QsPosToProposal: max: 0.655, avg: 0.426", "ConsensusProposalToOrdered: max: 0.340, avg: 0.321", "ConsensusOrderedToCommit: max: 0.537, avg: 0.499", "ConsensusProposalToCommit: max: 0.854, avg: 0.820"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.03s no progress at version 12596 (avg 0.23s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.36s no progress at version 2788432 (avg 8.36s) [limit 15].
Test Ok

github-actions[bot] avatar Aug 22 '24 21:08 github-actions[bot]

:white_check_mark: Forge suite compat success on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf

Compatibility test results for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf (PR)
1. Check liveness of validators at old version: d1bf834728a0cf166d993f4728dfca54f3086fb0
compatibility::simple-validator-upgrade::liveness-check : committed: 12549.03 txn/s, latency: 2543.15 ms, (p50: 1900 ms, p90: 4000 ms, p99: 18700 ms), latency samples: 442940
2. Upgrading first Validator to new version: 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 6651.70 txn/s, latency: 4237.88 ms, (p50: 4900 ms, p90: 5300 ms, p99: 5400 ms), latency samples: 120520
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6570.28 txn/s, latency: 4883.38 ms, (p50: 5000 ms, p90: 6900 ms, p99: 7100 ms), latency samples: 226600
3. Upgrading rest of first batch to new version: 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7209.38 txn/s, latency: 3930.32 ms, (p50: 4400 ms, p90: 4800 ms, p99: 4900 ms), latency samples: 134240
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7034.90 txn/s, latency: 4599.44 ms, (p50: 4900 ms, p90: 5300 ms, p99: 5400 ms), latency samples: 235460
4. upgrading second batch to new version: 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 523.03 txn/s, submitted: 711.31 txn/s, failed submission: 126.55 txn/s, expired: 188.28 txn/s, latency: 19304.23 ms, (p50: 1300 ms, p90: 59800 ms, p99: 66300 ms), latency samples: 56045
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 11413.95 txn/s, latency: 2704.10 ms, (p50: 2600 ms, p90: 4100 ms, p99: 5200 ms), latency samples: 371160
5. check swarm health
Compatibility test for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 8d86a1e889bbb2ad734e649346c6de0bda8fe5bf passed
Test Ok

github-actions[bot] avatar Aug 22 '24 21:08 github-actions[bot]

Forge is running suite framework_upgrade on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> b1fe922394dd739e6800d520debf15846a8fee34

github-actions[bot] avatar Aug 23 '24 21:08 github-actions[bot]

Forge is running suite realistic_env_max_load on b1fe922394dd739e6800d520debf15846a8fee34

github-actions[bot] avatar Aug 23 '24 21:08 github-actions[bot]

Forge is running suite compat on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> b1fe922394dd739e6800d520debf15846a8fee34

github-actions[bot] avatar Aug 23 '24 21:08 github-actions[bot]

:white_check_mark: Forge suite realistic_env_max_load success on b1fe922394dd739e6800d520debf15846a8fee34

two traffics test: inner traffic : committed: 12070.65 txn/s, latency: 3299.45 ms, (p50: 3000 ms, p90: 3900 ms, p99: 6900 ms), latency samples: 4589520
two traffics test : committed: 100.04 txn/s, latency: 3026.61 ms, (p50: 2600 ms, p90: 4400 ms, p99: 7300 ms), latency samples: 1840
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.268, avg: 0.227", "QsPosToProposal: max: 0.353, avg: 0.274", "ConsensusProposalToOrdered: max: 0.344, avg: 0.314", "ConsensusOrderedToCommit: max: 0.532, avg: 0.492", "ConsensusProposalToCommit: max: 0.852, avg: 0.806"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.86s no progress at version 14206 (avg 0.22s) [limit 15].
Max epoch-change gap was: 1 rounds at version 2509365 (avg 1.00) [limit 4], 9.01s no progress at version 2509365 (avg 9.01s) [limit 15].
Test Ok

github-actions[bot] avatar Aug 23 '24 21:08 github-actions[bot]

:white_check_mark: Forge suite compat success on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> b1fe922394dd739e6800d520debf15846a8fee34

Compatibility test results for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> b1fe922394dd739e6800d520debf15846a8fee34 (PR)
1. Check liveness of validators at old version: d1bf834728a0cf166d993f4728dfca54f3086fb0
compatibility::simple-validator-upgrade::liveness-check : committed: 10769.12 txn/s, latency: 3024.99 ms, (p50: 2100 ms, p90: 6500 ms, p99: 17800 ms), latency samples: 430240
2. Upgrading first Validator to new version: b1fe922394dd739e6800d520debf15846a8fee34
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7371.50 txn/s, latency: 3877.42 ms, (p50: 4200 ms, p90: 4800 ms, p99: 4900 ms), latency samples: 136980
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7409.64 txn/s, latency: 4348.71 ms, (p50: 4500 ms, p90: 6100 ms, p99: 6300 ms), latency samples: 248720
3. Upgrading rest of first batch to new version: b1fe922394dd739e6800d520debf15846a8fee34
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 7446.62 txn/s, latency: 3911.63 ms, (p50: 4300 ms, p90: 4700 ms, p99: 4800 ms), latency samples: 143140
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 6466.65 txn/s, latency: 4576.28 ms, (p50: 4800 ms, p90: 5100 ms, p99: 6100 ms), latency samples: 245600
4. upgrading second batch to new version: b1fe922394dd739e6800d520debf15846a8fee34
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 12008.96 txn/s, latency: 2251.18 ms, (p50: 2300 ms, p90: 2800 ms, p99: 3000 ms), latency samples: 209240
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10769.72 txn/s, latency: 2889.90 ms, (p50: 2600 ms, p90: 5800 ms, p99: 7800 ms), latency samples: 376820
5. check swarm health
Compatibility test for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> b1fe922394dd739e6800d520debf15846a8fee34 passed
Test Ok

github-actions[bot] avatar Aug 23 '24 21:08 github-actions[bot]

:white_check_mark: Forge suite framework_upgrade success on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> b1fe922394dd739e6800d520debf15846a8fee34

Compatibility test results for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> b1fe922394dd739e6800d520debf15846a8fee34 (PR)
Upgrade the nodes to version: b1fe922394dd739e6800d520debf15846a8fee34
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1253.16 txn/s, submitted: 1254.97 txn/s, failed submission: 1.81 txn/s, expired: 1.81 txn/s, latency: 2600.17 ms, (p50: 2400 ms, p90: 3900 ms, p99: 5700 ms), latency samples: 110660
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1140.59 txn/s, submitted: 1143.22 txn/s, failed submission: 2.63 txn/s, expired: 2.63 txn/s, latency: 2866.85 ms, (p50: 2400 ms, p90: 4900 ms, p99: 7200 ms), latency samples: 104120
5. check swarm health
Compatibility test for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> b1fe922394dd739e6800d520debf15846a8fee34 passed
Upgrade the remaining nodes to version: b1fe922394dd739e6800d520debf15846a8fee34
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 829.32 txn/s, submitted: 830.92 txn/s, failed submission: 1.60 txn/s, expired: 1.60 txn/s, latency: 4002.05 ms, (p50: 2900 ms, p90: 7500 ms, p99: 8800 ms), latency samples: 72580
Test Ok

github-actions[bot] avatar Aug 23 '24 21:08 github-actions[bot]

Forge is running suite compat on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 34accf5b01bf99f19518b97d3850bfff22fd8354

github-actions[bot] avatar Aug 24 '24 01:08 github-actions[bot]

Forge is running suite realistic_env_max_load on 34accf5b01bf99f19518b97d3850bfff22fd8354

github-actions[bot] avatar Aug 24 '24 01:08 github-actions[bot]

:white_check_mark: Forge suite compat success on d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 34accf5b01bf99f19518b97d3850bfff22fd8354

Compatibility test results for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 34accf5b01bf99f19518b97d3850bfff22fd8354 (PR)
1. Check liveness of validators at old version: d1bf834728a0cf166d993f4728dfca54f3086fb0
compatibility::simple-validator-upgrade::liveness-check : committed: 11512.88 txn/s, latency: 2789.78 ms, (p50: 2100 ms, p90: 6000 ms, p99: 12600 ms), latency samples: 415380
2. Upgrading first Validator to new version: 34accf5b01bf99f19518b97d3850bfff22fd8354
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7175.54 txn/s, latency: 3970.84 ms, (p50: 4400 ms, p90: 4900 ms, p99: 5200 ms), latency samples: 142200
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 7558.63 txn/s, latency: 4274.77 ms, (p50: 4400 ms, p90: 6500 ms, p99: 6800 ms), latency samples: 251420
3. Upgrading rest of first batch to new version: 34accf5b01bf99f19518b97d3850bfff22fd8354
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 4744.81 txn/s, latency: 5988.99 ms, (p50: 6800 ms, p90: 7600 ms, p99: 7700 ms), latency samples: 96980
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 4494.31 txn/s, latency: 7277.68 ms, (p50: 8000 ms, p90: 8300 ms, p99: 8600 ms), latency samples: 160280
4. upgrading second batch to new version: 34accf5b01bf99f19518b97d3850bfff22fd8354
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 12238.00 txn/s, latency: 2195.60 ms, (p50: 2300 ms, p90: 2700 ms, p99: 2900 ms), latency samples: 213060
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 7251.21 txn/s, latency: 3780.52 ms, (p50: 2500 ms, p90: 6100 ms, p99: 18400 ms), latency samples: 352500
5. check swarm health
Compatibility test for d1bf834728a0cf166d993f4728dfca54f3086fb0 ==> 34accf5b01bf99f19518b97d3850bfff22fd8354 passed
Test Ok

github-actions[bot] avatar Aug 24 '24 01:08 github-actions[bot]

:white_check_mark: Forge suite realistic_env_max_load success on 34accf5b01bf99f19518b97d3850bfff22fd8354

two traffics test: inner traffic : committed: 13105.46 txn/s, latency: 3036.33 ms, (p50: 2900 ms, p90: 3300 ms, p99: 10600 ms), latency samples: 4982960
two traffics test : committed: 100.05 txn/s, latency: 2807.34 ms, (p50: 2600 ms, p90: 3100 ms, p99: 10300 ms), latency samples: 1720
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.246, avg: 0.222", "QsPosToProposal: max: 0.283, avg: 0.217", "ConsensusProposalToOrdered: max: 0.329, avg: 0.297", "ConsensusOrderedToCommit: max: 0.494, avg: 0.464", "ConsensusProposalToCommit: max: 0.787, avg: 0.761"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 0.88s no progress at version 4718197 (avg 0.20s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 8.44s no progress at version 4718195 (avg 7.49s) [limit 15].
Test Ok

github-actions[bot] avatar Aug 24 '24 01:08 github-actions[bot]

Forge is running suite realistic_env_max_load on 09ca9f916c7fe408cda32bfe6dcf34263df29692

github-actions[bot] avatar Aug 24 '24 05:08 github-actions[bot]