debug pre-commit resend

[msmouse]

to run 100 nodes test

[trunk-io[bot]]

⏱️ 14h 28m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
forge-e2e-test / forge	7h 54m	🟥 🟥 🟥 🟥 🟥 (+3 more)
test-target-determinator	1h 41m	🟩 🟩 🟩 🟩 🟩 (+18 more)
general-lints	1h	🟩 🟩 🟩 🟩 🟩 (+14 more)
rust-cargo-deny	33m	🟩 🟩 🟩 🟩 🟩 (+14 more)
check-dynamic-deps	21m	🟩 🟩 🟩 🟩 🟩 (+16 more)
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
rust-move-tests	9m	🟩
semgrep/ci	9m	🟩 🟩 🟩 🟩 🟩 (+16 more)
rust-move-tests	9m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	8m	🟩
rust-move-tests	5m	🟥
file_change_determinator	4m	🟩 🟩 🟩 🟩 🟩 (+19 more)
file_change_determinator	4m	🟩 🟩 🟩 🟩 🟩 (+18 more)
file_change_determinator	4m	🟩 🟩 🟩 🟩 🟩 (+16 more)
permission-check	1m	🟩 🟩 🟩 🟩 🟩 (+20 more)
rust-move-tests	1m	⬜
permission-check	1m	🟩 🟩 🟩 🟩 🟩 (+20 more)
permission-check	1m	🟩 🟩 🟩 🟩 🟩 (+20 more)
permission-check	1m	🟩 🟩 🟩 🟩 🟩 (+20 more)
permission-check	1m	🟩 🟩 🟩 🟩 🟩 (+18 more)
rust-move-tests	57s	⬜
determine-docker-build-metadata	43s	🟩 🟩 🟩 🟩 🟩 (+18 more)
permission-check	7s	🟩 🟩
Backport PR	6s	🟥 🟥
rust-move-tests	1s	⬜

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
forge-e2e-test / forge	18m	14m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[github-actions[bot]]

Forge is running suite realistic_env_max_load on c0465773c38dd19e6845e069fbba4c526742188e

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on c0465773c38dd19e6845e069fbba4c526742188e

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:644"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724288670-c0465773c38dd19e6845e069fbba4c526","timestamp":"2024-08-22T05:03:18.608377Z","message":"Node validator-7 unhealthy: Unknown error error sending request for url (http://aptos-node-7-validator.forge-e2e-pr-14380.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:644"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724288670-c0465773c38dd19e6845e069fbba4c526","timestamp":"2024-08-22T05:03:28.683272Z","message":"Node validator-7 unhealthy: Unknown error error sending request for url (http://aptos-node-7-validator.forge-e2e-pr-14380.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:644"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724288670-c0465773c38dd19e6845e069fbba4c526","timestamp":"2024-08-22T05:03:38.712397Z","message":"Node validator-7 unhealthy: Unknown error error sending request for url (http://aptos-node-7-validator.forge-e2e-pr-14380.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:644"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724288670-c0465773c38dd19e6845e069fbba4c526","timestamp":"2024-08-22T05:03:48.750561Z","message":"Node validator-7 unhealthy: Unknown error error sending request for url (http://aptos-node-7-validator.forge-e2e-pr-14380.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:644"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724288670-c0465773c38dd19e6845e069fbba4c526","timestamp":"2024-08-22T05:03:58.825796Z","message":"Node validator-7 unhealthy: Unknown error error sending request for url (http://aptos-node-7-validator.forge-e2e-pr-14380.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:644"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724288670-c0465773c38dd19e6845e069fbba4c526","timestamp":"2024-08-22T05:04:08.857664Z","message":"Node validator-7 unhealthy: Unknown error error sending request for url (http://aptos-node-7-validator.forge-e2e-pr-14380.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:644"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724288670-c0465773c38dd19e6845e069fbba4c526","timestamp":"2024-08-22T05:04:18.895136Z","message":"Node validator-7 unhealthy: Unknown error error sending request for url (http://aptos-node-7-validator.forge-e2e-pr-14380.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:644"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724288670-c0465773c38dd19e6845e069fbba4c526","timestamp":"2024-08-22T05:04:28.956229Z","message":"Node validator-7 unhealthy: Unknown error error sending request for url (http://aptos-node-7-validator.forge-e2e-pr-14380.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:644"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724288670-c0465773c38dd19e6845e069fbba4c526","timestamp":"2024-08-22T05:04:38.988920Z","message":"Node validator-7 unhealthy: Unknown error error sending request for url (http://aptos-node-7-validator.forge-e2e-pr-14380.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:644"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724288670-c0465773c38dd19e6845e069fbba4c526","timestamp":"2024-08-22T05:04:49.036332Z","message":"Node validator-7 unhealthy: Unknown error error sending request for url (http://aptos-node-7-validator.forge-e2e-pr-14380.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
Debugging output:
Failed to get debugging output: error: You must be logged in to the server (Unauthorized)

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on c0465773c38dd19e6845e069fbba4c526742188e

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 7209d9ada62f674ff756a90c50630017fc402e1c

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on 7209d9ada62f674ff756a90c50630017fc402e1c

two traffics test: inner traffic : committed: 7910.45 txn/s, latency: 5015.76 ms, (p50: 3900 ms, p90: 9000 ms, p99: 13600 ms), latency samples: 3007720
Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 7910.447029407598, minimum TPS requirement 10000. Full stats: committed: 7910.45 txn/s, latency: 5015.76 ms, (p50: 3900 ms, p90: 9000 ms, p99: 13600 ms), latency samples: 3007720

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:453:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:506:9
   3: aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:257:9
   4: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:61:13
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   6: aptos_testcases::create_buffered_load::{{closure}}
             at ./testsuite/testcases/src/lib.rs:405:18
   7: <dyn aptos_testcases::NetworkLoadTest>::network_load_test::{{closure}}
             at ./testsuite/testcases/src/lib.rs:506:10
   8: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:267:14
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  10: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  12: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:628:37
  13: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  14: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  15: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  16: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  17: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  18: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  19: tokio::runtime::handle::Handle::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:310:22
  20: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  21: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:309:9
  22: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:611:49
  23: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:684:11
  24: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:611:30
  25: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
  26: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
  27: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  28: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  29: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  30: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  31: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  32: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  33: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  34: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  35: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start
Trailing Log Lines:
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724449232-7209d9ada62f674ff756a90c50630017f","timestamp":"2024-08-23T21:58:51.735410Z","message":"Deleting namespace forge-e2e-pr-14380: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724449232-7209d9ada62f674ff756a90c50630017f","timestamp":"2024-08-23T21:58:51.735436Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-14380"}

Failed to run tests:
Tests Failed
failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:636:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge191-0       1/1     Running     0          17m
aptos-node-0-validator-0                1/1     Running     0          16m
aptos-node-1-fullnode-eforge191-0       1/1     Running     0          17m
aptos-node-1-validator-0                1/1     Running     0          16m
aptos-node-10-validator-0               1/1     Running     0          16m
aptos-node-11-validator-0               1/1     Running     0          17m
aptos-node-12-validator-0               1/1     Running     0          17m
aptos-node-13-validator-0               1/1     Running     0          16m
aptos-node-14-validator-0               1/1     Running     0          17m
aptos-node-15-validator-0               1/1     Running     0          17m
aptos-node-16-validator-0               1/1     Running     0          16m
aptos-node-17-validator-0               1/1     Running     0          16m
aptos-node-18-validator-0               1/1     Running     0          17m
aptos-node-19-validator-0               1/1     Running     0          16m
aptos-node-2-fullnode-eforge191-0       1/1     Running     0          16m
aptos-node-2-validator-0                1/1     Running     0          17m
aptos-node-20-validator-0               1/1     Running     0          16m
aptos-node-21-validator-0               1/1     Running     0          17m
aptos-node-22-validator-0               1/1     Running     0          16m
aptos-node-23-validator-0               1/1     Running     0          16m
aptos-node-24-validator-0               1/1     Running     0          17m
aptos-node-25-validator-0               1/1     Running     0          16m
aptos-node-26-validator-0               1/1     Running     0          16m
aptos-node-27-validator-0               1/1     Running     0          16m
aptos-node-28-validator-0               1/1     Running     0          16m
aptos-node-29-validator-0               1/1     Running     0          16m
aptos-node-3-fullnode-eforge191-0       1/1     Running     0          16m
aptos-node-3-validator-0                1/1     Running     0          16m
aptos-node-30-validator-0               1/1     Running     0          16m
aptos-node-31-validator-0               1/1     Running     0          16m
aptos-node-32-validator-0               1/1     Running     0          16m
aptos-node-33-validator-0               1/1     Running     0          15m
aptos-node-34-validator-0               1/1     Running     0          16m
aptos-node-35-validator-0               1/1     Running     0          16m
aptos-node-36-validator-0               1/1     Running     0          16m
aptos-node-37-validator-0               1/1     Running     0          16m
aptos-node-38-validator-0               1/1     Running     0          16m
aptos-node-39-validator-0               1/1     Running     0          16m
aptos-node-4-fullnode-eforge191-0       1/1     Running     0          17m
aptos-node-4-validator-0                1/1     Running     0          16m
aptos-node-40-validator-0               1/1     Running     0          16m
aptos-node-41-validator-0               1/1     Running     0          16m
aptos-node-42-validator-0               1/1     Running     0          16m
aptos-node-43-validator-0               1/1     Running     0          16m
aptos-node-44-validator-0               1/1     Running     0          16m
aptos-node-45-validator-0               1/1     Running     0          16m
aptos-node-46-validator-0               1/1     Running     0          16m
aptos-node-47-validator-0               1/1     Running     0          16m
aptos-node-48-validator-0               1/1     Running     0          16m
aptos-node-49-validator-0               1/1     Running     0          16m
aptos-node-5-validator-0                1/1     Running     0          17m
aptos-node-50-validator-0               1/1     Running     0          16m
aptos-node-51-validator-0               1/1     Running     0          16m
aptos-node-52-validator-0               1/1     Running     0          16m
aptos-node-53-validator-0               1/1     Running     0          16m
aptos-node-54-validator-0               1/1     Running     0          16m
aptos-node-55-validator-0               1/1     Running     0          16m
aptos-node-56-validator-0               1/1     Running     0          16m
aptos-node-57-validator-0               1/1     Running     0          16m
aptos-node-58-validator-0               1/1     Running     0          16m
aptos-node-59-validator-0               1/1     Running     0          16m
aptos-node-6-validator-0                1/1     Running     0          17m
aptos-node-60-validator-0               1/1     Running     0          16m
aptos-node-61-validator-0               1/1     Running     0          17m
aptos-node-62-validator-0               1/1     Running     0          16m
aptos-node-63-validator-0               1/1     Running     0          16m
aptos-node-64-validator-0               1/1     Running     0          16m
aptos-node-65-validator-0               1/1     Running     0          16m
aptos-node-66-validator-0               1/1     Running     0          16m
aptos-node-67-validator-0               1/1     Running     0          17m
aptos-node-68-validator-0               1/1     Running     0          16m
aptos-node-69-validator-0               1/1     Running     0          17m
aptos-node-7-validator-0                1/1     Running     0          16m
aptos-node-70-validator-0               1/1     Running     0          16m
aptos-node-71-validator-0               1/1     Running     0          16m
aptos-node-72-validator-0               1/1     Running     0          16m
aptos-node-73-validator-0               1/1     Running     0          16m
aptos-node-74-validator-0               1/1     Running     0          16m
aptos-node-75-validator-0               1/1     Running     0          16m
aptos-node-76-validator-0               1/1     Running     0          16m
aptos-node-77-validator-0               1/1     Running     0          16m
aptos-node-78-validator-0               1/1     Running     0          16m
aptos-node-79-validator-0               1/1     Running     0          16m
aptos-node-8-validator-0                1/1     Running     0          17m
aptos-node-80-validator-0               1/1     Running     0          16m
aptos-node-81-validator-0               1/1     Running     0          16m
aptos-node-82-validator-0               1/1     Running     0          16m
aptos-node-83-validator-0               1/1     Running     0          16m
aptos-node-84-validator-0               1/1     Running     0          16m
aptos-node-85-validator-0               1/1     Running     0          16m
aptos-node-86-validator-0               1/1     Running     0          17m
aptos-node-87-validator-0               1/1     Running     0          16m
aptos-node-88-validator-0               1/1     Running     0          17m
aptos-node-89-validator-0               1/1     Running     0          16m
aptos-node-9-validator-0                1/1     Running     0          16m
aptos-node-90-validator-0               1/1     Running     0          16m
aptos-node-91-validator-0               1/1     Running     0          16m
aptos-node-92-validator-0               1/1     Running     0          16m
aptos-node-93-validator-0               1/1     Running     0          16m
aptos-node-94-validator-0               1/1     Running     0          16m
aptos-node-95-validator-0               1/1     Running     0          16m
aptos-node-96-validator-0               1/1     Running     0          16m
aptos-node-97-validator-0               1/1     Running     0          16m
aptos-node-98-validator-0               1/1     Running     0          16m
aptos-node-99-validator-0               1/1     Running     0          17m
genesis-aptos-genesis-eforge191-bxj29   0/1     Completed   0          17m

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on db6882302206111aa3b4d6b1a43763ed35734afd

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on db6882302206111aa3b4d6b1a43763ed35734afd

two traffics test: inner traffic : committed: 6739.91 txn/s, latency: 5865.04 ms, (p50: 4800 ms, p90: 10200 ms, p99: 14200 ms), latency samples: 2562660
Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 6739.909635653849, minimum TPS requirement 10000. Full stats: committed: 6739.91 txn/s, latency: 5865.04 ms, (p50: 4800 ms, p90: 10200 ms, p99: 14200 ms), latency samples: 2562660

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:453:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:506:9
   3: aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:257:9
   4: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:61:13
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   6: aptos_testcases::create_buffered_load::{{closure}}
             at ./testsuite/testcases/src/lib.rs:405:18
   7: <dyn aptos_testcases::NetworkLoadTest>::network_load_test::{{closure}}
             at ./testsuite/testcases/src/lib.rs:506:10
   8: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:267:14
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  10: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  12: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:628:37
  13: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  14: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  15: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  16: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  17: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  18: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  19: tokio::runtime::handle::Handle::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:310:22
  20: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  21: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:309:9
  22: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:611:49
  23: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:684:11
  24: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:611:30
  25: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
  26: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
  27: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  28: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  29: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  30: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  31: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  32: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  33: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  34: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  35: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start
Trailing Log Lines:
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724453412-db6882302206111aa3b4d6b1a43763ed3","timestamp":"2024-08-23T23:05:23.185197Z","message":"Deleting namespace forge-e2e-pr-14380: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724453412-db6882302206111aa3b4d6b1a43763ed3","timestamp":"2024-08-23T23:05:23.185228Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-14380"}

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:636:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                  READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge3-0       1/1     Running     0          14m
aptos-node-0-validator-0              1/1     Running     0          14m
aptos-node-1-fullnode-eforge3-0       1/1     Running     0          14m
aptos-node-1-validator-0              1/1     Running     0          14m
aptos-node-10-validator-0             1/1     Running     0          14m
aptos-node-11-validator-0             1/1     Running     0          14m
aptos-node-12-validator-0             1/1     Running     0          14m
aptos-node-13-validator-0             1/1     Running     0          14m
aptos-node-14-validator-0             1/1     Running     0          14m
aptos-node-15-validator-0             1/1     Running     0          14m
aptos-node-16-validator-0             1/1     Running     0          14m
aptos-node-17-validator-0             1/1     Running     0          14m
aptos-node-18-validator-0             1/1     Running     0          14m
aptos-node-19-validator-0             1/1     Running     0          14m
aptos-node-2-fullnode-eforge3-0       1/1     Running     0          14m
aptos-node-2-validator-0              1/1     Running     0          14m
aptos-node-20-validator-0             1/1     Running     0          14m
aptos-node-21-validator-0             1/1     Running     0          14m
aptos-node-22-validator-0             1/1     Running     0          14m
aptos-node-23-validator-0             1/1     Running     0          14m
aptos-node-24-validator-0             1/1     Running     0          14m
aptos-node-25-validator-0             1/1     Running     0          14m
aptos-node-26-validator-0             1/1     Running     0          14m
aptos-node-27-validator-0             1/1     Running     0          14m
aptos-node-28-validator-0             1/1     Running     0          14m
aptos-node-29-validator-0             1/1     Running     0          14m
aptos-node-3-fullnode-eforge3-0       1/1     Running     0          14m
aptos-node-3-validator-0              1/1     Running     0          14m
aptos-node-30-validator-0             1/1     Running     0          14m
aptos-node-31-validator-0             1/1     Running     0          14m
aptos-node-32-validator-0             1/1     Running     0          14m
aptos-node-33-validator-0             1/1     Running     0          14m
aptos-node-34-validator-0             1/1     Running     0          14m
aptos-node-35-validator-0             1/1     Running     0          14m
aptos-node-36-validator-0             1/1     Running     0          14m
aptos-node-37-validator-0             1/1     Running     0          14m
aptos-node-38-validator-0             1/1     Running     0          14m
aptos-node-39-validator-0             1/1     Running     0          14m
aptos-node-4-fullnode-eforge3-0       1/1     Running     0          14m
aptos-node-4-validator-0              1/1     Running     0          14m
aptos-node-40-validator-0             1/1     Running     0          14m
aptos-node-41-validator-0             1/1     Running     0          14m
aptos-node-42-validator-0             1/1     Running     0          14m
aptos-node-43-validator-0             1/1     Running     0          14m
aptos-node-44-validator-0             1/1     Running     0          14m
aptos-node-45-validator-0             1/1     Running     0          14m
aptos-node-46-validator-0             1/1     Running     0          14m
aptos-node-47-validator-0             1/1     Running     0          14m
aptos-node-48-validator-0             1/1     Running     0          14m
aptos-node-49-validator-0             1/1     Running     0          14m
aptos-node-5-validator-0              1/1     Running     0          14m
aptos-node-50-validator-0             1/1     Running     0          14m
aptos-node-51-validator-0             1/1     Running     0          14m
aptos-node-52-validator-0             1/1     Running     0          14m
aptos-node-53-validator-0             1/1     Running     0          14m
aptos-node-54-validator-0             1/1     Running     0          14m
aptos-node-55-validator-0             1/1     Running     0          14m
aptos-node-56-validator-0             1/1     Running     0          14m
aptos-node-57-validator-0             1/1     Running     0          14m
aptos-node-58-validator-0             1/1     Running     0          14m
aptos-node-59-validator-0             1/1     Running     0          14m
aptos-node-6-validator-0              1/1     Running     0          14m
aptos-node-60-validator-0             1/1     Running     0          14m
aptos-node-61-validator-0             1/1     Running     0          14m
aptos-node-62-validator-0             1/1     Running     0          14m
aptos-node-63-validator-0             1/1     Running     0          14m
aptos-node-64-validator-0             1/1     Running     0          14m
aptos-node-65-validator-0             1/1     Running     0          14m
aptos-node-66-validator-0             1/1     Running     0          14m
aptos-node-67-validator-0             1/1     Running     0          14m
aptos-node-68-validator-0             1/1     Running     0          14m
aptos-node-69-validator-0             1/1     Running     0          14m
aptos-node-7-validator-0              1/1     Running     0          14m
aptos-node-70-validator-0             1/1     Running     0          14m
aptos-node-71-validator-0             1/1     Running     0          14m
aptos-node-72-validator-0             1/1     Running     0          14m
aptos-node-73-validator-0             1/1     Running     0          14m
aptos-node-74-validator-0             1/1     Running     0          14m
aptos-node-75-validator-0             1/1     Running     0          14m
aptos-node-76-validator-0             1/1     Running     0          14m
aptos-node-77-validator-0             1/1     Running     0          14m
aptos-node-78-validator-0             1/1     Running     0          14m
aptos-node-79-validator-0             1/1     Running     0          14m
aptos-node-8-validator-0              1/1     Running     0          14m
aptos-node-80-validator-0             1/1     Running     0          14m
aptos-node-81-validator-0             1/1     Running     0          14m
aptos-node-82-validator-0             1/1     Running     0          14m
aptos-node-83-validator-0             1/1     Running     0          14m
aptos-node-84-validator-0             1/1     Running     0          14m
aptos-node-85-validator-0             1/1     Running     0          14m
aptos-node-86-validator-0             1/1     Running     0          14m
aptos-node-87-validator-0             1/1     Running     0          14m
aptos-node-88-validator-0             1/1     Running     0          14m
aptos-node-89-validator-0             1/1     Running     0          14m
aptos-node-9-validator-0              1/1     Running     0          14m
aptos-node-90-validator-0             1/1     Running     0          14m
aptos-node-91-validator-0             1/1     Running     0          14m
aptos-node-92-validator-0             1/1     Running     0          14m
aptos-node-93-validator-0             1/1     Running     0          14m
aptos-node-94-validator-0             1/1     Running     0          14m
aptos-node-95-validator-0             1/1     Running     0          14m
aptos-node-96-validator-0             1/1     Running     0          14m
aptos-node-97-validator-0             1/1     Running     0          14m
aptos-node-98-validator-0             1/1     Running     0          14m
aptos-node-99-validator-0             1/1     Running     0          14m
genesis-aptos-genesis-eforge3-d9gjl   0/1     Completed   0          14m

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on ac9c295281cfc4ba398ce36be23b255d7f8f20fd

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on ac9c295281cfc4ba398ce36be23b255d7f8f20fd

two traffics test: inner traffic : committed: 8727.61 txn/s, latency: 4565.10 ms, (p50: 3600 ms, p90: 8100 ms, p99: 13700 ms), latency samples: 3318400
Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 8727.6094839018, minimum TPS requirement 10000. Full stats: committed: 8727.61 txn/s, latency: 4565.10 ms, (p50: 3600 ms, p90: 8100 ms, p99: 13700 ms), latency samples: 3318400

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:453:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:506:9
   3: aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:257:9
   4: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:61:13
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   6: aptos_testcases::create_buffered_load::{{closure}}
             at ./testsuite/testcases/src/lib.rs:405:18
   7: <dyn aptos_testcases::NetworkLoadTest>::network_load_test::{{closure}}
             at ./testsuite/testcases/src/lib.rs:506:10
   8: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:267:14
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  10: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  12: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:628:37
  13: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  14: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  15: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  16: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  17: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  18: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  19: tokio::runtime::handle::Handle::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:310:22
  20: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  21: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:309:9
  22: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:611:49
  23: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:684:11
  24: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:611:30
  25: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
  26: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
  27: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  28: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  29: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  30: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  31: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  32: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  33: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  34: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  35: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start
Trailing Log Lines:
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724463662-ac9c295281cfc4ba398ce36be23b255d7","timestamp":"2024-08-24T01:54:32.859647Z","message":"Deleting namespace forge-e2e-pr-14380: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724463662-ac9c295281cfc4ba398ce36be23b255d7","timestamp":"2024-08-24T01:54:32.859672Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-14380"}

failures:
Failed to run tests:
Tests Failed
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:636:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge28-0       1/1     Running     0          12m
aptos-node-0-validator-0               1/1     Running     0          12m
aptos-node-1-fullnode-eforge28-0       1/1     Running     0          12m
aptos-node-1-validator-0               1/1     Running     0          12m
aptos-node-10-validator-0              1/1     Running     0          12m
aptos-node-11-validator-0              1/1     Running     0          12m
aptos-node-12-validator-0              1/1     Running     0          12m
aptos-node-13-validator-0              1/1     Running     0          12m
aptos-node-14-validator-0              1/1     Running     0          12m
aptos-node-15-validator-0              1/1     Running     0          12m
aptos-node-16-validator-0              1/1     Running     0          12m
aptos-node-17-validator-0              1/1     Running     0          12m
aptos-node-18-validator-0              1/1     Running     0          12m
aptos-node-19-validator-0              1/1     Running     0          12m
aptos-node-2-fullnode-eforge28-0       1/1     Running     0          12m
aptos-node-2-validator-0               1/1     Running     0          12m
aptos-node-20-validator-0              1/1     Running     0          12m
aptos-node-21-validator-0              1/1     Running     0          12m
aptos-node-22-validator-0              1/1     Running     0          12m
aptos-node-23-validator-0              1/1     Running     0          12m
aptos-node-24-validator-0              1/1     Running     0          12m
aptos-node-25-validator-0              1/1     Running     0          12m
aptos-node-26-validator-0              1/1     Running     0          12m
aptos-node-27-validator-0              1/1     Running     0          12m
aptos-node-28-validator-0              1/1     Running     0          12m
aptos-node-29-validator-0              1/1     Running     0          12m
aptos-node-3-fullnode-eforge28-0       1/1     Running     0          12m
aptos-node-3-validator-0               1/1     Running     0          12m
aptos-node-30-validator-0              1/1     Running     0          12m
aptos-node-31-validator-0              1/1     Running     0          12m
aptos-node-32-validator-0              1/1     Running     0          12m
aptos-node-33-validator-0              1/1     Running     0          12m
aptos-node-34-validator-0              1/1     Running     0          12m
aptos-node-35-validator-0              1/1     Running     0          12m
aptos-node-36-validator-0              1/1     Running     0          12m
aptos-node-37-validator-0              1/1     Running     0          12m
aptos-node-38-validator-0              1/1     Running     0          12m
aptos-node-39-validator-0              1/1     Running     0          12m
aptos-node-4-fullnode-eforge28-0       1/1     Running     0          12m
aptos-node-4-validator-0               1/1     Running     0          12m
aptos-node-40-validator-0              1/1     Running     0          12m
aptos-node-41-validator-0              1/1     Running     0          12m
aptos-node-42-validator-0              1/1     Running     0          12m
aptos-node-43-validator-0              1/1     Running     0          12m
aptos-node-44-validator-0              1/1     Running     0          12m
aptos-node-45-validator-0              1/1     Running     0          12m
aptos-node-46-validator-0              1/1     Running     0          12m
aptos-node-47-validator-0              1/1     Running     0          12m
aptos-node-48-validator-0              1/1     Running     0          12m
aptos-node-49-validator-0              1/1     Running     0          12m
aptos-node-5-validator-0               1/1     Running     0          12m
aptos-node-50-validator-0              1/1     Running     0          12m
aptos-node-51-validator-0              1/1     Running     0          12m
aptos-node-52-validator-0              1/1     Running     0          12m
aptos-node-53-validator-0              1/1     Running     0          12m
aptos-node-54-validator-0              1/1     Running     0          12m
aptos-node-55-validator-0              1/1     Running     0          12m
aptos-node-56-validator-0              1/1     Running     0          12m
aptos-node-57-validator-0              1/1     Running     0          12m
aptos-node-58-validator-0              1/1     Running     0          12m
aptos-node-59-validator-0              1/1     Running     0          12m
aptos-node-6-validator-0               1/1     Running     0          12m
aptos-node-60-validator-0              1/1     Running     0          12m
aptos-node-61-validator-0              1/1     Running     0          12m
aptos-node-62-validator-0              1/1     Running     0          12m
aptos-node-63-validator-0              1/1     Running     0          12m
aptos-node-64-validator-0              1/1     Running     0          12m
aptos-node-65-validator-0              1/1     Running     0          12m
aptos-node-66-validator-0              1/1     Running     0          12m
aptos-node-67-validator-0              1/1     Running     0          12m
aptos-node-68-validator-0              1/1     Running     0          12m
aptos-node-69-validator-0              1/1     Running     0          12m
aptos-node-7-validator-0               1/1     Running     0          12m
aptos-node-70-validator-0              1/1     Running     0          12m
aptos-node-71-validator-0              1/1     Running     0          12m
aptos-node-72-validator-0              1/1     Running     0          12m
aptos-node-73-validator-0              1/1     Running     0          12m
aptos-node-74-validator-0              1/1     Running     0          12m
aptos-node-75-validator-0              1/1     Running     0          12m
aptos-node-76-validator-0              1/1     Running     0          12m
aptos-node-77-validator-0              1/1     Running     0          12m
aptos-node-78-validator-0              1/1     Running     0          12m
aptos-node-79-validator-0              1/1     Running     0          12m
aptos-node-8-validator-0               1/1     Running     0          12m
aptos-node-80-validator-0              1/1     Running     0          12m
aptos-node-81-validator-0              1/1     Running     0          12m
aptos-node-82-validator-0              1/1     Running     0          12m
aptos-node-83-validator-0              1/1     Running     0          12m
aptos-node-84-validator-0              1/1     Running     0          12m
aptos-node-85-validator-0              1/1     Running     0          12m
aptos-node-86-validator-0              1/1     Running     0          12m
aptos-node-87-validator-0              1/1     Running     0          12m
aptos-node-88-validator-0              1/1     Running     0          12m
aptos-node-89-validator-0              1/1     Running     0          12m
aptos-node-9-validator-0               1/1     Running     0          12m
aptos-node-90-validator-0              1/1     Running     0          12m
aptos-node-91-validator-0              1/1     Running     0          12m
aptos-node-92-validator-0              1/1     Running     0          12m
aptos-node-93-validator-0              1/1     Running     0          12m
aptos-node-94-validator-0              1/1     Running     0          12m
aptos-node-95-validator-0              1/1     Running     0          12m
aptos-node-96-validator-0              1/1     Running     0          12m
aptos-node-97-validator-0              1/1     Running     0          12m
aptos-node-98-validator-0              1/1     Running     0          12m
aptos-node-99-validator-0              1/1     Running     0          12m
genesis-aptos-genesis-eforge28-5qt7p   0/1     Completed   0          13m

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 49f189185ab200067689920f234ecb8f7c63a8ad

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on 49f189185ab200067689920f234ecb8f7c63a8ad

two traffics test: inner traffic : committed: 7007.65 txn/s, latency: 5669.65 ms, (p50: 4500 ms, p90: 9900 ms, p99: 14900 ms), latency samples: 2664460
Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 7007.6528763172, minimum TPS requirement 10000. Full stats: committed: 7007.65 txn/s, latency: 5669.65 ms, (p50: 4500 ms, p90: 9900 ms, p99: 14900 ms), latency samples: 2664460

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:453:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:506:9
   3: aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:257:9
   4: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:61:13
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   6: aptos_testcases::create_buffered_load::{{closure}}
             at ./testsuite/testcases/src/lib.rs:405:18
   7: <dyn aptos_testcases::NetworkLoadTest>::network_load_test::{{closure}}
             at ./testsuite/testcases/src/lib.rs:506:10
   8: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:267:14
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  10: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  12: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:628:37
  13: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  14: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  15: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  16: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  17: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  18: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  19: tokio::runtime::handle::Handle::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:310:22
  20: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  21: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:309:9
  22: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:611:49
  23: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:684:11
  24: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:611:30
  25: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
  26: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
  27: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  28: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  29: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  30: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  31: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  32: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  33: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  34: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  35: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start
Trailing Log Lines:
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724476011-49f189185ab200067689920f234ecb8f7","timestamp":"2024-08-24T05:21:36.184312Z","message":"Deleting namespace forge-e2e-pr-14380: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724476011-49f189185ab200067689920f234ecb8f7","timestamp":"2024-08-24T05:21:36.184338Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-14380"}
Failed to run tests:
Tests Failed

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:636:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge177-0       1/1     Running     0          13m
aptos-node-0-validator-0                1/1     Running     0          13m
aptos-node-1-fullnode-eforge177-0       1/1     Running     0          13m
aptos-node-1-validator-0                1/1     Running     0          13m
aptos-node-10-validator-0               1/1     Running     0          13m
aptos-node-11-validator-0               1/1     Running     0          13m
aptos-node-12-validator-0               1/1     Running     0          13m
aptos-node-13-validator-0               1/1     Running     0          13m
aptos-node-14-validator-0               1/1     Running     0          13m
aptos-node-15-validator-0               1/1     Running     0          13m
aptos-node-16-validator-0               1/1     Running     0          13m
aptos-node-17-validator-0               1/1     Running     0          13m
aptos-node-18-validator-0               1/1     Running     0          13m
aptos-node-19-validator-0               1/1     Running     0          13m
aptos-node-2-fullnode-eforge177-0       1/1     Running     0          13m
aptos-node-2-validator-0                1/1     Running     0          13m
aptos-node-20-validator-0               1/1     Running     0          13m
aptos-node-21-validator-0               1/1     Running     0          13m
aptos-node-22-validator-0               1/1     Running     0          13m
aptos-node-23-validator-0               1/1     Running     0          13m
aptos-node-24-validator-0               1/1     Running     0          13m
aptos-node-25-validator-0               1/1     Running     0          13m
aptos-node-26-validator-0               1/1     Running     0          13m
aptos-node-27-validator-0               1/1     Running     0          13m
aptos-node-28-validator-0               1/1     Running     0          13m
aptos-node-29-validator-0               1/1     Running     0          13m
aptos-node-3-fullnode-eforge177-0       1/1     Running     0          13m
aptos-node-3-validator-0                1/1     Running     0          13m
aptos-node-30-validator-0               1/1     Running     0          13m
aptos-node-31-validator-0               1/1     Running     0          13m
aptos-node-32-validator-0               1/1     Running     0          13m
aptos-node-33-validator-0               1/1     Running     0          13m
aptos-node-34-validator-0               1/1     Running     0          13m
aptos-node-35-validator-0               1/1     Running     0          13m
aptos-node-36-validator-0               1/1     Running     0          13m
aptos-node-37-validator-0               1/1     Running     0          13m
aptos-node-38-validator-0               1/1     Running     0          13m
aptos-node-39-validator-0               1/1     Running     0          13m
aptos-node-4-fullnode-eforge177-0       1/1     Running     0          13m
aptos-node-4-validator-0                1/1     Running     0          13m
aptos-node-40-validator-0               1/1     Running     0          13m
aptos-node-41-validator-0               1/1     Running     0          13m
aptos-node-42-validator-0               1/1     Running     0          13m
aptos-node-43-validator-0               1/1     Running     0          13m
aptos-node-44-validator-0               1/1     Running     0          13m
aptos-node-45-validator-0               1/1     Running     0          13m
aptos-node-46-validator-0               1/1     Running     0          13m
aptos-node-47-validator-0               1/1     Running     0          13m
aptos-node-48-validator-0               1/1     Running     0          13m
aptos-node-49-validator-0               1/1     Running     0          13m
aptos-node-5-validator-0                1/1     Running     0          13m
aptos-node-50-validator-0               1/1     Running     0          13m
aptos-node-51-validator-0               1/1     Running     0          13m
aptos-node-52-validator-0               1/1     Running     0          13m
aptos-node-53-validator-0               1/1     Running     0          13m
aptos-node-54-validator-0               1/1     Running     0          13m
aptos-node-55-validator-0               1/1     Running     0          13m
aptos-node-56-validator-0               1/1     Running     0          13m
aptos-node-57-validator-0               1/1     Running     0          13m
aptos-node-58-validator-0               1/1     Running     0          13m
aptos-node-59-validator-0               1/1     Running     0          13m
aptos-node-6-validator-0                1/1     Running     0          13m
aptos-node-60-validator-0               1/1     Running     0          13m
aptos-node-61-validator-0               1/1     Running     0          13m
aptos-node-62-validator-0               1/1     Running     0          13m
aptos-node-63-validator-0               1/1     Running     0          13m
aptos-node-64-validator-0               1/1     Running     0          13m
aptos-node-65-validator-0               1/1     Running     0          13m
aptos-node-66-validator-0               1/1     Running     0          13m
aptos-node-67-validator-0               1/1     Running     0          13m
aptos-node-68-validator-0               1/1     Running     0          13m
aptos-node-69-validator-0               1/1     Running     0          13m
aptos-node-7-validator-0                1/1     Running     0          13m
aptos-node-70-validator-0               1/1     Running     0          13m
aptos-node-71-validator-0               1/1     Running     0          13m
aptos-node-72-validator-0               1/1     Running     0          13m
aptos-node-73-validator-0               1/1     Running     0          13m
aptos-node-74-validator-0               1/1     Running     0          13m
aptos-node-75-validator-0               1/1     Running     0          13m
aptos-node-76-validator-0               1/1     Running     0          13m
aptos-node-77-validator-0               1/1     Running     0          13m
aptos-node-78-validator-0               1/1     Running     0          13m
aptos-node-79-validator-0               1/1     Running     0          13m
aptos-node-8-validator-0                1/1     Running     0          13m
aptos-node-80-validator-0               1/1     Running     0          13m
aptos-node-81-validator-0               1/1     Running     0          13m
aptos-node-82-validator-0               1/1     Running     0          13m
aptos-node-83-validator-0               1/1     Running     0          13m
aptos-node-84-validator-0               1/1     Running     0          13m
aptos-node-85-validator-0               1/1     Running     0          13m
aptos-node-86-validator-0               1/1     Running     0          13m
aptos-node-87-validator-0               1/1     Running     0          13m
aptos-node-88-validator-0               1/1     Running     0          13m
aptos-node-89-validator-0               1/1     Running     0          13m
aptos-node-9-validator-0                1/1     Running     0          13m
aptos-node-90-validator-0               1/1     Running     0          13m
aptos-node-91-validator-0               1/1     Running     0          13m
aptos-node-92-validator-0               1/1     Running     0          13m
aptos-node-93-validator-0               1/1     Running     0          13m
aptos-node-94-validator-0               1/1     Running     0          13m
aptos-node-95-validator-0               1/1     Running     0          13m
aptos-node-96-validator-0               1/1     Running     0          13m
aptos-node-97-validator-0               1/1     Running     0          13m
aptos-node-98-validator-0               1/1     Running     0          13m
aptos-node-99-validator-0               1/1     Running     0          13m
genesis-aptos-genesis-eforge177-sb79k   0/1     Completed   0          14m

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 2aac3efa877aa6740dab3b348dcaf26665cf5345

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on 2aac3efa877aa6740dab3b348dcaf26665cf5345

two traffics test: inner traffic : committed: 6913.11 txn/s, submitted: 6916.72 txn/s, expired: 3.61 txn/s, latency: 5604.95 ms, (p50: 4200 ms, p90: 9300 ms, p99: 15400 ms), latency samples: 2628508
Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 6913.113862183231, minimum TPS requirement 10000. Full stats: committed: 6913.11 txn/s, submitted: 6916.72 txn/s, expired: 3.61 txn/s, latency: 5604.95 ms, (p50: 4200 ms, p90: 9300 ms, p99: 15400 ms), latency samples: 2628508

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::success_criteria::SuccessCriteriaChecker::check_tps
             at ./testsuite/forge/src/success_criteria.rs:453:13
   2: aptos_forge::success_criteria::SuccessCriteriaChecker::check_throughput
             at ./testsuite/forge/src/success_criteria.rs:506:9
   3: aptos_forge::success_criteria::SuccessCriteriaChecker::check_core_for_success
             at ./testsuite/forge/src/success_criteria.rs:257:9
   4: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_testcases::NetworkLoadTest>::test::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:61:13
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
   6: aptos_testcases::create_buffered_load::{{closure}}
             at ./testsuite/testcases/src/lib.rs:405:18
   7: <dyn aptos_testcases::NetworkLoadTest>::network_load_test::{{closure}}
             at ./testsuite/testcases/src/lib.rs:506:10
   8: <dyn aptos_testcases::NetworkLoadTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:267:14
   9: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  10: <aptos_testcases::two_traffics_test::TwoTrafficsTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/two_traffics_test.rs:77:47
  11: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  12: <aptos_testcases::CompositeNetworkTest as aptos_forge::interface::network::NetworkTest>::run::{{closure}}
             at ./testsuite/testcases/src/lib.rs:628:37
  13: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/future/future.rs:123:9
  14: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  15: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  16: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  17: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  18: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  19: tokio::runtime::handle::Handle::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:310:22
  20: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  21: tokio::runtime::handle::Handle::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/handle.rs:309:9
  22: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:611:49
  23: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:684:11
  24: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:611:30
  25: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
  26: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
  27: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
  28: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
  29: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
  30: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
  31: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  32: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  33: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  34: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  35: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start
Trailing Log Lines:
  36: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  37: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  38: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  39: main
  40: __libc_start_main
  41: _start


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724694867-2aac3efa877aa6740dab3b348dcaf2666","timestamp":"2024-08-26T18:10:16.555630Z","message":"Deleting namespace forge-e2e-pr-14380: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-14380-1724694867-2aac3efa877aa6740dab3b348dcaf2666","timestamp":"2024-08-26T18:10:16.555662Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-14380"}

failures:
    CompositeNetworkTest

Failed to run tests:
Tests Failed
test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:636:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:453:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:379:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/sys_common/backtrace.rs:155:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:166:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panic.rs:146:14
  15: std::rt::lang_start_internal
             at /rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/rt.rs:148:20
  16: main
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge105-0       1/1     Running     0          14m
aptos-node-0-validator-0                1/1     Running     0          14m
aptos-node-1-fullnode-eforge105-0       1/1     Running     0          14m
aptos-node-1-validator-0                1/1     Running     0          14m
aptos-node-10-validator-0               1/1     Running     0          14m
aptos-node-11-validator-0               1/1     Running     0          14m
aptos-node-12-validator-0               1/1     Running     0          14m
aptos-node-13-validator-0               1/1     Running     0          14m
aptos-node-14-validator-0               1/1     Running     0          14m
aptos-node-15-validator-0               1/1     Running     0          14m
aptos-node-16-validator-0               1/1     Running     0          14m
aptos-node-17-validator-0               1/1     Running     0          14m
aptos-node-18-validator-0               1/1     Running     0          14m
aptos-node-19-validator-0               1/1     Running     0          14m
aptos-node-2-fullnode-eforge105-0       1/1     Running     0          14m
aptos-node-2-validator-0                1/1     Running     0          14m
aptos-node-20-validator-0               1/1     Running     0          14m
aptos-node-21-validator-0               1/1     Running     0          14m
aptos-node-22-validator-0               1/1     Running     0          14m
aptos-node-23-validator-0               1/1     Running     0          14m
aptos-node-24-validator-0               1/1     Running     0          14m
aptos-node-25-validator-0               1/1     Running     0          14m
aptos-node-26-validator-0               1/1     Running     0          14m
aptos-node-27-validator-0               1/1     Running     0          14m
aptos-node-28-validator-0               1/1     Running     0          14m
aptos-node-29-validator-0               1/1     Running     0          14m
aptos-node-3-fullnode-eforge105-0       1/1     Running     0          14m
aptos-node-3-validator-0                1/1     Running     0          14m
aptos-node-30-validator-0               1/1     Running     0          14m
aptos-node-31-validator-0               1/1     Running     0          14m
aptos-node-32-validator-0               1/1     Running     0          14m
aptos-node-33-validator-0               1/1     Running     0          14m
aptos-node-34-validator-0               1/1     Running     0          14m
aptos-node-35-validator-0               1/1     Running     0          14m
aptos-node-36-validator-0               1/1     Running     0          14m
aptos-node-37-validator-0               1/1     Running     0          14m
aptos-node-38-validator-0               1/1     Running     0          14m
aptos-node-39-validator-0               1/1     Running     0          14m
aptos-node-4-fullnode-eforge105-0       1/1     Running     0          14m
aptos-node-4-validator-0                1/1     Running     0          14m
aptos-node-40-validator-0               1/1     Running     0          14m
aptos-node-41-validator-0               1/1     Running     0          14m
aptos-node-42-validator-0               1/1     Running     0          14m
aptos-node-43-validator-0               1/1     Running     0          14m
aptos-node-44-validator-0               1/1     Running     0          14m
aptos-node-45-validator-0               1/1     Running     0          14m
aptos-node-46-validator-0               1/1     Running     0          14m
aptos-node-47-validator-0               1/1     Running     0          14m
aptos-node-48-validator-0               1/1     Running     0          14m
aptos-node-49-validator-0               1/1     Running     0          14m
aptos-node-5-validator-0                1/1     Running     0          14m
aptos-node-50-validator-0               1/1     Running     0          14m
aptos-node-51-validator-0               1/1     Running     0          14m
aptos-node-52-validator-0               1/1     Running     0          14m
aptos-node-53-validator-0               1/1     Running     0          14m
aptos-node-54-validator-0               1/1     Running     0          14m
aptos-node-55-validator-0               1/1     Running     0          14m
aptos-node-56-validator-0               1/1     Running     0          14m
aptos-node-57-validator-0               1/1     Running     0          14m
aptos-node-58-validator-0               1/1     Running     0          14m
aptos-node-59-validator-0               1/1     Running     0          14m
aptos-node-6-validator-0                1/1     Running     0          14m
aptos-node-60-validator-0               1/1     Running     0          14m
aptos-node-61-validator-0               1/1     Running     0          14m
aptos-node-62-validator-0               1/1     Running     0          14m
aptos-node-63-validator-0               1/1     Running     0          14m
aptos-node-64-validator-0               1/1     Running     0          14m
aptos-node-65-validator-0               1/1     Running     0          14m
aptos-node-66-validator-0               1/1     Running     0          14m
aptos-node-67-validator-0               1/1     Running     0          14m
aptos-node-68-validator-0               1/1     Running     0          14m
aptos-node-69-validator-0               1/1     Running     0          14m
aptos-node-7-validator-0                1/1     Running     0          14m
aptos-node-70-validator-0               1/1     Running     0          14m
aptos-node-71-validator-0               1/1     Running     0          14m
aptos-node-72-validator-0               1/1     Running     0          14m
aptos-node-73-validator-0               1/1     Running     0          14m
aptos-node-74-validator-0               1/1     Running     0          14m
aptos-node-75-validator-0               1/1     Running     0          14m
aptos-node-76-validator-0               1/1     Running     0          14m
aptos-node-77-validator-0               1/1     Running     0          14m
aptos-node-78-validator-0               1/1     Running     0          14m
aptos-node-79-validator-0               1/1     Running     0          14m
aptos-node-8-validator-0                1/1     Running     0          14m
aptos-node-80-validator-0               1/1     Running     0          14m
aptos-node-81-validator-0               1/1     Running     0          14m
aptos-node-82-validator-0               1/1     Running     0          14m
aptos-node-83-validator-0               1/1     Running     0          14m
aptos-node-84-validator-0               1/1     Running     0          14m
aptos-node-85-validator-0               1/1     Running     0          14m
aptos-node-86-validator-0               1/1     Running     0          14m
aptos-node-87-validator-0               1/1     Running     0          14m
aptos-node-88-validator-0               1/1     Running     0          14m
aptos-node-89-validator-0               1/1     Running     0          14m
aptos-node-9-validator-0                1/1     Running     0          14m
aptos-node-90-validator-0               1/1     Running     0          14m
aptos-node-91-validator-0               1/1     Running     0          14m
aptos-node-92-validator-0               1/1     Running     0          14m
aptos-node-93-validator-0               1/1     Running     0          14m
aptos-node-94-validator-0               1/1     Running     0          14m
aptos-node-95-validator-0               1/1     Running     0          14m
aptos-node-96-validator-0               1/1     Running     0          14m
aptos-node-97-validator-0               1/1     Running     0          14m
aptos-node-98-validator-0               1/1     Running     0          14m
aptos-node-99-validator-0               1/1     Running     0          14m
genesis-aptos-genesis-eforge105-6dvfl   0/1     Completed   0          15m

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking