[observer] initial commit

[zekun000]

trafficstars

Description

Type of Change

• [x] New feature
• [ ] Bug fix
• [ ] Breaking change
• [ ] Performance improvement
• [ ] Refactoring
• [ ] Dependency update
• [ ] Documentation update
• [ ] Tests

Which Components or Systems Does This Change Impact?

• [x] Validator Node
• [x] Full Node (API, Indexer, etc.)
• [ ] Move/Aptos Virtual Machine
• [ ] Aptos Framework
• [ ] Aptos CLI/SDK
• [ ] Developer Infrastructure
• [ ] Other (specify)

How Has This Been Tested?

Key Areas to Review

Checklist

• [ ] I have read and followed the CONTRIBUTING doc
• [ ] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I identified and added all stakeholders and component owners affected by this change as reviewers
• [ ] I tested both happy and unhappy path of the functionality
• [ ] I have made corresponding changes to the documentation

[trunk-io[bot]]

⏱️ 23h 57m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-smoke-tests	4h 25m	🟩 🟥 🟩 ⬜ 🟩 (+5 more)
execution-performance / single-node-performance	3h 39m	🟩 🟩 🟩 🟩 🟩 (+5 more)
forge-e2e-test / forge	2h 55m	🟥 🟥 🟥 🟩 🟩 (+3 more)
rust-images / rust-all	2h 16m	🟩 🟥 🟩 ⬜ 🟩 (+5 more)
rust-targeted-unit-tests	2h 2m	🟥 🟥 🟥 ⬜ 🟩 (+5 more)
forge-compat-test / forge	1h 34m	🟩 🟩 🟩 🟩 🟩 (+3 more)
rust-move-tests	1h 30m	🟩 🟩 🟩 ⬜ 🟩 (+6 more)
cli-e2e-tests / run-cli-tests	1h 7m	🟩 🟩 🟩 🟩 🟩 (+3 more)
rust-lints	54m	🟩 🟥 🟩 🟥 🟩 (+5 more)
run-tests-main-branch	46m	🟩 🟩 🟩 🟩 🟩 (+5 more)
rust-build-cached-packages	42m	🟩 🟩 🟩 🟩 🟩 (+5 more)
check	41m	🟩 🟩 🟩 🟩 🟩 (+5 more)
check-dynamic-deps	21m	🟩 🟩 🟩 🟩 🟩 (+6 more)
general-lints	21m	🟩 🟩 🟩 🟩 🟩 (+5 more)
indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose	19m	🟩 🟩 🟩 🟩 🟩 (+3 more)
node-api-compatibility-tests / node-api-compatibility-tests	7m	🟩 🟩 🟩 🟩 🟩 (+3 more)
semgrep/ci	4m	🟩 🟩 🟩 🟩 🟩 (+6 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+6 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+6 more)
execution-performance / file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+5 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+5 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+4 more)
permission-check	38s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	35s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	32s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	30s	🟩 🟩 🟩 🟩 🟩 (+6 more)
permission-check	29s	🟩 🟩 🟩 🟩 🟩 (+5 more)
determine-docker-build-metadata	18s	🟩 🟩 🟩 🟩 🟩 (+5 more)

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
rust-move-tests	3m	11m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 2fd98aa4c309028366c88928a9690440fd7541c7

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on aptos-node-v1.10.1 ==> 2fd98aa4c309028366c88928a9690440fd7541c7

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on aptos-node-v1.10.1 ==> 2fd98aa4c309028366c88928a9690440fd7541c7

Compatibility test results for aptos-node-v1.10.1 ==> 2fd98aa4c309028366c88928a9690440fd7541c7 (PR)
1. Check liveness of validators at old version: aptos-node-v1.10.1
compatibility::simple-validator-upgrade::liveness-check : committed: 7070 txn/s, latency: 4687 ms, (p50: 5000 ms, p90: 6800 ms, p99: 8000 ms), latency samples: 247480
2. Upgrading first Validator to new version: 2fd98aa4c309028366c88928a9690440fd7541c7
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1882 txn/s, latency: 15477 ms, (p50: 19900 ms, p90: 21800 ms, p99: 22200 ms), latency samples: 92220
3. Upgrading rest of first batch to new version: 2fd98aa4c309028366c88928a9690440fd7541c7
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1676 txn/s, latency: 17123 ms, (p50: 19900 ms, p90: 21900 ms, p99: 22300 ms), latency samples: 83840
4. upgrading second batch to new version: 2fd98aa4c309028366c88928a9690440fd7541c7
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3686 txn/s, latency: 8707 ms, (p50: 9900 ms, p90: 12100 ms, p99: 12600 ms), latency samples: 143760
5. check swarm health
Compatibility test for aptos-node-v1.10.1 ==> 2fd98aa4c309028366c88928a9690440fd7541c7 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on aptos-node-v1.10.1 ==> 08d82484b066dd3462d0c870193c0b05ae698fe9

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 08d82484b066dd3462d0c870193c0b05ae698fe9

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 22f8bc3b9de19b0f5c612d8824f2bf80993c391f

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on aptos-node-v1.10.1 ==> 22f8bc3b9de19b0f5c612d8824f2bf80993c391f

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on aptos-node-v1.10.1 ==> 22f8bc3b9de19b0f5c612d8824f2bf80993c391f

Compatibility test results for aptos-node-v1.10.1 ==> 22f8bc3b9de19b0f5c612d8824f2bf80993c391f (PR)
1. Check liveness of validators at old version: aptos-node-v1.10.1
compatibility::simple-validator-upgrade::liveness-check : committed: 5911 txn/s, latency: 5246 ms, (p50: 4800 ms, p90: 9600 ms, p99: 15700 ms), latency samples: 230560
2. Upgrading first Validator to new version: 22f8bc3b9de19b0f5c612d8824f2bf80993c391f
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1780 txn/s, latency: 16148 ms, (p50: 18700 ms, p90: 22300 ms, p99: 22600 ms), latency samples: 92600
3. Upgrading rest of first batch to new version: 22f8bc3b9de19b0f5c612d8824f2bf80993c391f
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1718 txn/s, latency: 15642 ms, (p50: 19600 ms, p90: 22000 ms, p99: 22600 ms), latency samples: 91100
4. upgrading second batch to new version: 22f8bc3b9de19b0f5c612d8824f2bf80993c391f
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3711 txn/s, latency: 8639 ms, (p50: 9900 ms, p90: 12000 ms, p99: 12500 ms), latency samples: 144740
5. check swarm health
Compatibility test for aptos-node-v1.10.1 ==> 22f8bc3b9de19b0f5c612d8824f2bf80993c391f passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on 22f8bc3b9de19b0f5c612d8824f2bf80993c391f

two traffics test: inner traffic : committed: 7936 txn/s, latency: 4949 ms, (p50: 4800 ms, p90: 5700 ms, p99: 11100 ms), latency samples: 3420600
two traffics test : committed: 100 txn/s, latency: 1941 ms, (p50: 1800 ms, p90: 2200 ms, p99: 6900 ms), latency samples: 1740
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.219, avg: 0.203", "QsPosToProposal: max: 0.286, avg: 0.252", "ConsensusProposalToOrdered: max: 0.466, avg: 0.430", "ConsensusOrderedToCommit: max: 0.341, avg: 0.312", "ConsensusProposalToCommit: max: 0.773, avg: 0.742"]
Max round gap was 1 [limit 4] at version 1720394. Max no progress secs was 4.62489 [limit 15] at version 1720394.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[davidiw]

wen issue or aip?

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 2eda0fb37dbb4c9476fb916fe518b67f130d8911

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on aptos-node-v1.10.1 ==> 2eda0fb37dbb4c9476fb916fe518b67f130d8911

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on aptos-node-v1.10.1 ==> 2eda0fb37dbb4c9476fb916fe518b67f130d8911

Compatibility test results for aptos-node-v1.10.1 ==> 2eda0fb37dbb4c9476fb916fe518b67f130d8911 (PR)
1. Check liveness of validators at old version: aptos-node-v1.10.1
compatibility::simple-validator-upgrade::liveness-check : committed: 6608 txn/s, latency: 5024 ms, (p50: 4800 ms, p90: 8700 ms, p99: 9600 ms), latency samples: 231280
2. Upgrading first Validator to new version: 2eda0fb37dbb4c9476fb916fe518b67f130d8911
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1660 txn/s, latency: 17646 ms, (p50: 19400 ms, p90: 24400 ms, p99: 24700 ms), latency samples: 89660
3. Upgrading rest of first batch to new version: 2eda0fb37dbb4c9476fb916fe518b67f130d8911
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1836 txn/s, latency: 15761 ms, (p50: 18400 ms, p90: 22000 ms, p99: 22500 ms), latency samples: 91800
4. upgrading second batch to new version: 2eda0fb37dbb4c9476fb916fe518b67f130d8911
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3662 txn/s, latency: 8726 ms, (p50: 9800 ms, p90: 12600 ms, p99: 12900 ms), latency samples: 142840
5. check swarm health
Compatibility test for aptos-node-v1.10.1 ==> 2eda0fb37dbb4c9476fb916fe518b67f130d8911 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on 2eda0fb37dbb4c9476fb916fe518b67f130d8911

Forge test runner terminated:
Trailing Log Lines:
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  15: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:552:29
  16: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  17: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  18: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Debugging output:
Failed to get debugging output: error: You must be logged in to the server (Unauthorized)

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on aptos-node-v1.10.1 ==> 12512529fe167702af3b35fa1e1b751bc96a1487

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 12512529fe167702af3b35fa1e1b751bc96a1487

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on 12512529fe167702af3b35fa1e1b751bc96a1487

Forge test runner terminated:
Trailing Log Lines:
  29: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start
Debugging output:
NAME                                   READY   STATUS             RESTARTS      AGE
aptos-node-0-fullnode-eforge78-0       0/1     CrashLoopBackOff   2 (16s ago)   2m33s
aptos-node-0-validator-0               1/1     Running            0             2m33s
aptos-node-1-fullnode-eforge78-0       0/1     Error              2 (20s ago)   2m33s
aptos-node-1-validator-0               1/1     Running            0             2m33s
aptos-node-2-fullnode-eforge78-0       0/1     CrashLoopBackOff   2 (20s ago)   2m34s
aptos-node-2-validator-0               1/1     Running            0             2m34s
aptos-node-3-fullnode-eforge78-0       0/1     Error              1 (18s ago)   2m32s
aptos-node-3-validator-0               1/1     Running            0             2m34s
aptos-node-4-fullnode-eforge78-0       0/1     Error              2 (23s ago)   2m34s
aptos-node-4-validator-0               1/1     Running            0             2m32s
aptos-node-5-validator-0               1/1     Running            0             2m34s
aptos-node-6-validator-0               1/1     Running            0             2m33s
genesis-aptos-genesis-eforge78-88vrc   0/1     Completed          0             2m55s

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on aptos-node-v1.10.1 ==> 12512529fe167702af3b35fa1e1b751bc96a1487

Compatibility test results for aptos-node-v1.10.1 ==> 12512529fe167702af3b35fa1e1b751bc96a1487 (PR)
1. Check liveness of validators at old version: aptos-node-v1.10.1
compatibility::simple-validator-upgrade::liveness-check : committed: 6890 txn/s, latency: 4802 ms, (p50: 4800 ms, p90: 7500 ms, p99: 9300 ms), latency samples: 241160
2. Upgrading first Validator to new version: 12512529fe167702af3b35fa1e1b751bc96a1487
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1540 txn/s, latency: 18854 ms, (p50: 23300 ms, p90: 30400 ms, p99: 30700 ms), latency samples: 78560
3. Upgrading rest of first batch to new version: 12512529fe167702af3b35fa1e1b751bc96a1487
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1584 txn/s, latency: 17736 ms, (p50: 19500 ms, p90: 25000 ms, p99: 25600 ms), latency samples: 87160
4. upgrading second batch to new version: 12512529fe167702af3b35fa1e1b751bc96a1487
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2556 txn/s, latency: 12074 ms, (p50: 11400 ms, p90: 19700 ms, p99: 21300 ms), latency samples: 102260
5. check swarm health
Compatibility test for aptos-node-v1.10.1 ==> 12512529fe167702af3b35fa1e1b751bc96a1487 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 0668022d953466865e95adcee33e42fa963589c8

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on aptos-node-v1.10.1 ==> 0668022d953466865e95adcee33e42fa963589c8

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on 0668022d953466865e95adcee33e42fa963589c8

Test Failed: test NetworkLoadTest

Caused by:
    Failed to create accounts: Account 0x05a9b877a84d94032b8793f6c2cc78eb440df23ac91bff9506cf1821c31b3d04 couldn't mint
    
    Caused by:
        0: Tried executing 10 txns, request counters: "success 1930, failed submit [530, 140, 40, 40, 10], failed wait [590, 160, 70, 40, 10, 10], by client: [(460, 20, 0): http://aptos-node-2-fullnode.forge-e2e-pr-12863.svc:8080/v1/]\n[(550, 20, 0): http://aptos-node-1-fullnode.forge-e2e-pr-12863.svc:8080/v1/]\n[(370, 0, 160): http://aptos-node-0-fullnode.forge-e2e-pr-12863.svc:8080/v1/]\n[(0, 720, 720): http://aptos-node-3-fullnode.forge-e2e-pr-12863.svc:8080/v1/]"
        1: Unknown error Ledger on endpoint (http://aptos-node-3-fullnode.forge-e2e-pr-12863.svc:8080/v1/) is more than 60s behind current time, timing out waiting for the transaction. Warning, transaction (c977c14d) might still succeed.
    
    Stack backtrace:
       0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
       1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
       2: aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter::submit_check_and_retry::{{closure}}
                 at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:127:28
       3: <futures_util::future::maybe_done::MaybeDone<Fut> as core::future::future::Future>::poll
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/maybe_done.rs:95:38
       4: <futures_util::future::join_all::JoinAll<F> as core::future::future::Future>::poll
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/join_all.rs:143:24
       5: <aptos_transaction_emitter_lib::emitter::transaction_executor::RestApiReliableTransactionSubmitter as aptos_transaction_generator_lib::ReliableTransactionSubmitter>::execute_transactions_with_counter::{{closure}}
                 at ./crates/transaction-emitter-lib/src/emitter/transaction_executor.rs:309:10
       6: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
       7: aptos_transaction_emitter_lib::emitter::account_minter::create_and_fund_new_accounts::{{closure}}
                 at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:548:14
       8: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/futures_unordered/mod.rs:518:17
       9: futures_util::stream::stream::StreamExt::poll_next_unpin
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/mod.rs:1638:9
      10: <futures_util::stream::stream::buffer_unordered::BufferUnordered<St> as futures_core::stream::Stream>::poll_next
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/buffer_unordered.rs:75:38
      11: <futures_util::stream::stream::collect::Collect<St,C> as core::future::future::Future>::poll
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/stream/stream/collect.rs:50:26
      12: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_accounts::{{closure}}
                 at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:378:14
      13: aptos_transaction_emitter_lib::emitter::create_accounts::{{closure}}
                 at ./crates/transaction-emitter-lib/src/emitter/mod.rs:1195:14
      14: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
                 at ./crates/transaction-emitter-lib/src/emitter/mod.rs:717:10
      15: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
                 at ./crates/transaction-emitter-lib/src/emitter/mod.rs:827:14
      16: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
                 at ./crates/transaction-emitter-lib/src/emitter/mod.rs:859:14
      17: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
      18: tokio::runtime::coop::with_budget
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
      19: tokio::runtime::coop::budget
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
      20: tokio::runtime::park::CachedParkThread::block_on
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
      21: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
      22: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
      23: tokio::runtime::context::runtime::enter_runtime
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
      24: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
      25: tokio::runtime::runtime::Runtime::block_on
                 at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
      26: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
                 at ./testsuite/testcases/src/two_traffics_test.rs:52:21
      27: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
                 at ./testsuite/testcases/src/lib.rs:311:13
      28: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
                 at ./testsuite/testcases/src/lib.rs:180:30
      29: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
                 at ./testsuite/testcases/src/lib.rs:499:9
      30: aptos_forge::runner::Forge<F>::run::{{closure}}
                 at ./testsuite/forge/src/runner.rs:598:42
      31: aptos_forge::runner::run_test
                 at ./testsuite/forge/src/runner.rs:666:11
      32: aptos_forge::runner::Forge<F>::run
                 at ./testsuite/forge/src/runner.rs:598:30
      33: forge::run_forge
                 at ./testsuite/forge-cli/src/main.rs:427:11
      34: forge::main
                 at ./testsuite/forge-cli/src/main.rs:353:21
      35: core::ops::function::FnOnce::call_once
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
      36: std::sys_common::backtrace::__rust_begin_short_backtrace
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
      37: std::rt::lang_start::{{closure}}
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
      38: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
      39: std::panicking::try::do_call
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
      40: std::panicking::try
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
      41: std::panic::catch_unwind
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
      42: std::rt::lang_start_internal::{{closure}}
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
      43: std::panicking::try::do_call
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
      44: std::panicking::try
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
      45: std::panic::catch_unwind
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
      46: std::rt::lang_start_internal
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
      47: std::rt::lang_start
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
      48: __libc_start_main
      49: _start

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_accounts::{{closure}}::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:381:26
   2: core::result::Result<T,E>::map_err
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:829:27
   3: aptos_transaction_emitter_lib::emitter::account_minter::AccountMinter::create_and_fund_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/account_minter.rs:376:25
   4: aptos_transaction_emitter_lib::emitter::create_accounts::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:1195:14
   5: aptos_transaction_emitter_lib::emitter::TxnEmitter::start_job::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:717:10
   6: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for_impl::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:827:14
   7: aptos_transaction_emitter_lib::emitter::TxnEmitter::emit_txn_for::{{closure}}
             at ./crates/transaction-emitter-lib/src/emitter/mod.rs:859:14
   8: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   9: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  10: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  11: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  12: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  13: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  14: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  15: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  16: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  17: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test
             at ./testsuite/testcases/src/two_traffics_test.rs:52:21
  18: <dyn aptos_testcases::NetworkLoadTest>::network_load_test
             at ./testsuite/testcases/src/lib.rs:311:13
  19: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:180:30
  20: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/lib.rs:499:9
  21: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  22: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  23: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  24: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  25: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  26: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  27: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  28: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  29: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  30: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  31: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  32: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  33: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  34: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  35: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  36: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  37: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  38: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  39: __libc_start_main
  40: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  36: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  37: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  38: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  39: __libc_start_main
  40: _start


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-12863-1713068530-0668022d953466865e95adcee33e42fa9","timestamp":"2024-04-14T04:30:50.156356Z","message":"Deleting namespace forge-e2e-pr-12863: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-12863-1713068530-0668022d953466865e95adcee33e42fa9","timestamp":"2024-04-14T04:30:50.156377Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-12863"}
Failed to run tests:
Tests Failed

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge75-0       1/1     Running     0          7m57s
aptos-node-0-validator-0               1/1     Running     0          7m57s
aptos-node-1-fullnode-eforge75-0       1/1     Running     0          7m57s
aptos-node-1-validator-0               1/1     Running     0          7m56s
aptos-node-2-fullnode-eforge75-0       1/1     Running     0          7m57s
aptos-node-2-validator-0               1/1     Running     0          7m56s
aptos-node-3-fullnode-eforge75-0       1/1     Running     0          7m57s
aptos-node-3-validator-0               1/1     Running     0          7m57s
aptos-node-4-fullnode-eforge75-0       1/1     Running     0          7m57s
aptos-node-4-validator-0               1/1     Running     0          7m57s
aptos-node-5-validator-0               1/1     Running     0          7m57s
aptos-node-6-validator-0               1/1     Running     0          7m57s
genesis-aptos-genesis-eforge75-sk97f   0/1     Completed   0          8m13s

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on aptos-node-v1.10.1 ==> 0668022d953466865e95adcee33e42fa963589c8

Compatibility test results for aptos-node-v1.10.1 ==> 0668022d953466865e95adcee33e42fa963589c8 (PR)
1. Check liveness of validators at old version: aptos-node-v1.10.1
compatibility::simple-validator-upgrade::liveness-check : committed: 6720 txn/s, latency: 4919 ms, (p50: 4800 ms, p90: 8000 ms, p99: 9400 ms), latency samples: 235200
2. Upgrading first Validator to new version: 0668022d953466865e95adcee33e42fa963589c8
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1796 txn/s, latency: 16318 ms, (p50: 18600 ms, p90: 22600 ms, p99: 22800 ms), latency samples: 93440
3. Upgrading rest of first batch to new version: 0668022d953466865e95adcee33e42fa963589c8
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1498 txn/s, latency: 19075 ms, (p50: 21500 ms, p90: 27300 ms, p99: 28300 ms), latency samples: 77940
4. upgrading second batch to new version: 0668022d953466865e95adcee33e42fa963589c8
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3471 txn/s, latency: 8623 ms, (p50: 9900 ms, p90: 11600 ms, p99: 12000 ms), latency samples: 145820
5. check swarm health
Compatibility test for aptos-node-v1.10.1 ==> 0668022d953466865e95adcee33e42fa963589c8 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on aptos-node-v1.10.1 ==> d4e57074dd79f1668991d8d27bd128cb6e5f42b4

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on d4e57074dd79f1668991d8d27bd128cb6e5f42b4

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on aptos-node-v1.10.1 ==> d4e57074dd79f1668991d8d27bd128cb6e5f42b4

Compatibility test results for aptos-node-v1.10.1 ==> d4e57074dd79f1668991d8d27bd128cb6e5f42b4 (PR)
1. Check liveness of validators at old version: aptos-node-v1.10.1
compatibility::simple-validator-upgrade::liveness-check : committed: 6270 txn/s, latency: 5287 ms, (p50: 4800 ms, p90: 9300 ms, p99: 13000 ms), latency samples: 219480
2. Upgrading first Validator to new version: d4e57074dd79f1668991d8d27bd128cb6e5f42b4
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1713 txn/s, latency: 16512 ms, (p50: 19800 ms, p90: 22800 ms, p99: 23900 ms), latency samples: 89080
3. Upgrading rest of first batch to new version: d4e57074dd79f1668991d8d27bd128cb6e5f42b4
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1779 txn/s, latency: 16189 ms, (p50: 19400 ms, p90: 23400 ms, p99: 23800 ms), latency samples: 88960
4. upgrading second batch to new version: d4e57074dd79f1668991d8d27bd128cb6e5f42b4
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3595 txn/s, latency: 8688 ms, (p50: 9700 ms, p90: 12400 ms, p99: 12600 ms), latency samples: 143820
5. check swarm health
Compatibility test for aptos-node-v1.10.1 ==> d4e57074dd79f1668991d8d27bd128cb6e5f42b4 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on d4e57074dd79f1668991d8d27bd128cb6e5f42b4

two traffics test: inner traffic : committed: 6877 txn/s, submitted: 6880 txn/s, expired: 3 txn/s, latency: 5107 ms, (p50: 5100 ms, p90: 6000 ms, p99: 10500 ms), latency samples: 3301200
two traffics test : committed: 100 txn/s, latency: 1357 ms, (p50: 1300 ms, p90: 1600 ms, p99: 1700 ms), latency samples: 2000
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.205, avg: 0.202", "QsPosToProposal: max: 0.290, avg: 0.265", "ConsensusProposalToOrdered: max: 0.494, avg: 0.444", "ConsensusOrderedToCommit: max: 0.334, avg: 0.312", "ConsensusProposalToCommit: max: 0.776, avg: 0.756"]
Max round gap was 1 [limit 4] at version 1610275. Max no progress secs was 4.524944 [limit 15] at version 1610275.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 6b4013c9d1b85eaf5cfcae7484d5b5c7aee3a5a1

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on aptos-node-v1.10.1 ==> 6b4013c9d1b85eaf5cfcae7484d5b5c7aee3a5a1

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on aptos-node-v1.10.1 ==> 6b4013c9d1b85eaf5cfcae7484d5b5c7aee3a5a1

Compatibility test results for aptos-node-v1.10.1 ==> 6b4013c9d1b85eaf5cfcae7484d5b5c7aee3a5a1 (PR)
1. Check liveness of validators at old version: aptos-node-v1.10.1
compatibility::simple-validator-upgrade::liveness-check : committed: 6397 txn/s, latency: 5180 ms, (p50: 4800 ms, p90: 8900 ms, p99: 14400 ms), latency samples: 223920
2. Upgrading first Validator to new version: 6b4013c9d1b85eaf5cfcae7484d5b5c7aee3a5a1
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1767 txn/s, latency: 16673 ms, (p50: 19000 ms, p90: 22600 ms, p99: 22800 ms), latency samples: 91900
3. Upgrading rest of first batch to new version: 6b4013c9d1b85eaf5cfcae7484d5b5c7aee3a5a1
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1489 txn/s, latency: 15821 ms, (p50: 19500 ms, p90: 22500 ms, p99: 23100 ms), latency samples: 90880
4. upgrading second batch to new version: 6b4013c9d1b85eaf5cfcae7484d5b5c7aee3a5a1
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3619 txn/s, latency: 8877 ms, (p50: 9600 ms, p90: 12600 ms, p99: 12900 ms), latency samples: 144760
5. check swarm health
Compatibility test for aptos-node-v1.10.1 ==> 6b4013c9d1b85eaf5cfcae7484d5b5c7aee3a5a1 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking