add final pepper derivation to the pepper service

[heliuchuan]

Description

Type of Change

• [ ] New feature
• [ ] Bug fix
• [ ] Breaking change
• [ ] Performance improvement
• [ ] Refactoring
• [ ] Dependency update
• [ ] Documentation update
• [ ] Tests

Which Components or Systems Does This Change Impact?

• [ ] Validator Node
• [ ] Full Node (API, Indexer, etc.)
• [ ] Move/Aptos Virtual Machine
• [ ] Aptos Framework
• [ ] Aptos CLI/SDK
• [ ] Developer Infrastructure
• [ ] Other (specify)

How Has This Been Tested?

Key Areas to Review

Checklist

• [ ] I have read and followed the CONTRIBUTING doc
• [ ] I have performed a self-review of my own code
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I identified and added all stakeholders and component owners affected by this change as reviewers
• [ ] I tested both happy and unhappy path of the functionality
• [ ] I have made corresponding changes to the documentation

[trunk-io[bot]]

⏱️ 3h 37m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-targeted-unit-tests	1h 41m	🟩 ⬜ 🟩 ⬜ 🟩 (+1 more)
rust-move-tests	49m	🟩 ⬜ 🟩 ⬜ 🟩 (+1 more)
run-tests-main-branch	27m	🟩 🟩 🟩 🟩 🟩 (+1 more)
rust-lints	17m	🟥 ⬜ 🟥 🟥 🟥 (+1 more)
general-lints	10m	🟩 🟩 🟩 🟩 🟩 (+1 more)
check-dynamic-deps	8m	🟩 🟩 🟩 🟩 🟩 (+1 more)
semgrep/ci	2m	🟩 🟩 🟩 🟩 🟩 (+1 more)
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+1 more)
file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+1 more)
permission-check	18s	🟩 🟩 🟩 🟩 🟩 (+1 more)
permission-check	16s	🟩 🟩 🟩 🟩 🟩 (+1 more)
permission-check	16s	🟩 🟩 🟩 🟩 🟩 (+1 more)
permission-check	14s	🟩 🟩 🟩 🟩 🟩 (+1 more)

🚨 1 job on the last run was significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
rust-targeted-unit-tests	24m	16m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

[zjma]

why does it feel like it is no longer just pepper service any more?

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> cc456b9b9958c7920c320024470880d323cb07ce

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> cc456b9b9958c7920c320024470880d323cb07ce

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on cc456b9b9958c7920c320024470880d323cb07ce

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> cc456b9b9958c7920c320024470880d323cb07ce

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on cc456b9b9958c7920c320024470880d323cb07ce

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> cc456b9b9958c7920c320024470880d323cb07ce

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite realistic_env_max_load failure on cc456b9b9958c7920c320024470880d323cb07ce

Forge test runner terminated:
Trailing Log Lines:
  29: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  32: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  33: __libc_start_main
  34: _start
Debugging output:
NAME                                   READY   STATUS            RESTARTS   AGE
aptos-node-0-fullnode-eforge39-0       0/1     PodInitializing   0          114s
aptos-node-0-validator-0               0/1     PodInitializing   0          114s
aptos-node-1-fullnode-eforge39-0       0/1     Pending           0          114s
aptos-node-1-validator-0               0/1     Pending           0          113s
aptos-node-2-fullnode-eforge39-0       0/1     PodInitializing   0          114s
aptos-node-2-validator-0               0/1     PodInitializing   0          114s
aptos-node-3-fullnode-eforge39-0       0/1     Pending           0          113s
aptos-node-3-validator-0               0/1     ErrImagePull      0          114s
aptos-node-4-fullnode-eforge39-0       0/1     Pending           0          113s
aptos-node-4-validator-0               0/1     PodInitializing   0          114s
aptos-node-5-validator-0               1/1     Running           0          114s
aptos-node-6-validator-0               1/1     Running           0          114s
genesis-aptos-genesis-eforge39-dgx66   0/1     Completed         0          2m19s

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> cc456b9b9958c7920c320024470880d323cb07ce

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> cc456b9b9958c7920c320024470880d323cb07ce (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 5947.940156653716 txn/s, latency: 5187.606828973496 ms, (p50: 4800 ms, p90: 9600 ms, p99: 14400 ms), latency samples: 231660
2. Upgrading first Validator to new version: cc456b9b9958c7920c320024470880d323cb07ce
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1782.797221808105 txn/s, latency: 16321.776635111875 ms, (p50: 18700 ms, p90: 23200 ms, p99: 23500 ms), latency samples: 92960
3. Upgrading rest of first batch to new version: cc456b9b9958c7920c320024470880d323cb07ce
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1485.2996137561902 txn/s, latency: 18451.88597940613 ms, (p50: 18500 ms, p90: 29500 ms, p99: 29900 ms), latency samples: 83520
4. upgrading second batch to new version: cc456b9b9958c7920c320024470880d323cb07ce
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3419.323595173715 txn/s, latency: 8677.796302162236 ms, (p50: 9900 ms, p90: 11400 ms, p99: 12400 ms), latency samples: 145220
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> cc456b9b9958c7920c320024470880d323cb07ce passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on cc456b9b9958c7920c320024470880d323cb07ce

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> cc456b9b9958c7920c320024470880d323cb07ce

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on cc456b9b9958c7920c320024470880d323cb07ce

two traffics test: inner traffic : committed: 8361.542240861763 txn/s, latency: 4690.629321719382 ms, (p50: 4500 ms, p90: 5400 ms, p99: 9900 ms), latency samples: 3610600
two traffics test : committed: 100.03785374375414 txn/s, latency: 1935.8968085106383 ms, (p50: 1900 ms, p90: 2200 ms, p99: 6200 ms), latency samples: 1880
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.209, avg: 0.202", "QsPosToProposal: max: 0.218, avg: 0.206", "ConsensusProposalToOrdered: max: 0.432, avg: 0.403", "ConsensusOrderedToCommit: max: 0.396, avg: 0.383", "ConsensusProposalToCommit: max: 0.800, avg: 0.785"]
Max round gap was 1 [limit 4] at version 1715874. Max no progress secs was 4.814996 [limit 15] at version 1715874.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite framework_upgrade failure on 01b24e7e3548382dd25440b39a0438a993387f12 ==> cc456b9b9958c7920c320024470880d323cb07ce

Forge test runner terminated:
Trailing Log Lines:
  27: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  28: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  29: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  30: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  31: __libc_start_main
  32: _start
Debugging output:

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite realistic_env_max_load on 2466418b71d3b1a750390feef760bb56962f6077

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite compat on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite compat success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077 (PR)
1. Check liveness of validators at old version: 01b24e7e3548382dd25440b39a0438a993387f12
compatibility::simple-validator-upgrade::liveness-check : committed: 5677.6533753724625 txn/s, latency: 5194.934623031652 ms, (p50: 4800 ms, p90: 8400 ms, p99: 11100 ms), latency samples: 251480
2. Upgrading first Validator to new version: 2466418b71d3b1a750390feef760bb56962f6077
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1670.204311052461 txn/s, latency: 17005.953250547846 ms, (p50: 19900 ms, p90: 26000 ms, p99: 26600 ms), latency samples: 82140
3. Upgrading rest of first batch to new version: 2466418b71d3b1a750390feef760bb56962f6077
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1432.9805438919168 txn/s, latency: 19813.23591101695 ms, (p50: 22100 ms, p90: 29600 ms, p99: 31900 ms), latency samples: 75520
4. upgrading second batch to new version: 2466418b71d3b1a750390feef760bb56962f6077
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3021.1844879004684 txn/s, latency: 10380.511800526663 ms, (p50: 11700 ms, p90: 12600 ms, p99: 12900 ms), latency samples: 121520
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077 passed
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite realistic_env_max_load success on 2466418b71d3b1a750390feef760bb56962f6077

two traffics test: inner traffic : committed: 7811.244261215844 txn/s, latency: 5009.684693569667 ms, (p50: 4800 ms, p90: 5700 ms, p99: 10800 ms), latency samples: 3380540
two traffics test : committed: 100.037995266203 txn/s, latency: 1971.453488372093 ms, (p50: 1900 ms, p90: 2200 ms, p99: 5900 ms), latency samples: 1720
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.206, avg: 0.201", "QsPosToProposal: max: 0.254, avg: 0.233", "ConsensusProposalToOrdered: max: 0.480, avg: 0.435", "ConsensusOrderedToCommit: max: 0.401, avg: 0.382", "ConsensusProposalToCommit: max: 0.830, avg: 0.817"]
Max round gap was 1 [limit 4] at version 1620840. Max no progress secs was 4.643848 [limit 15] at version 1620840.
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite framework_upgrade failure on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077 (PR)
Upgrade the nodes to version: 2466418b71d3b1a750390feef760bb56962f6077
Test Failed: Retry due to pod aptos-node-0-validator-0 status PodStatus { conditions: Some([PodCondition { last_probe_time: None, last_transition_time: Some(Time(2024-05-10T00:15:44Z)), message: Some("0/23 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) had volume node affinity conflict, 21 Insufficient cpu, 21 Insufficient memory. preemption: 0/23 nodes are available: 2 Preemption is not helpful for scheduling, 21 No preemption victims found for incoming pod.."), reason: Some("Unschedulable"), status: "False", type_: "PodScheduled" }]), container_statuses: None, ephemeral_container_statuses: None, host_ip: None, init_container_statuses: None, message: None, nominated_node_name: None, phase: Some("Pending"), pod_ip: None, pod_ips: None, qos_class: Some("Burstable"), reason: None, start_time: None }

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_forge::backend::k8s::stateful_set::wait_stateful_set::{{closure}}
             at ./testsuite/forge/src/backend/k8s/stateful_set.rs:69:5
   3: aptos_forge::backend::k8s::stateful_set::scale_stateful_set_replicas::{{closure}}
             at ./testsuite/forge/src/backend/k8s/stateful_set.rs:243:6
   4: <aptos_forge::backend::k8s::node::K8sNode as aptos_forge::interface::node::Node>::start::{{closure}}
             at ./testsuite/forge/src/backend/k8s/node.rs:137:84
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
   6: <aptos_forge::backend::k8s::swarm::K8sSwarm as aptos_forge::interface::swarm::Swarm>::upgrade_validator::{{closure}}
             at ./testsuite/forge/src/backend/k8s/swarm.rs:270:27
   7: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
   8: aptos_testcases::batch_update::{{closure}}
             at ./testsuite/testcases/src/lib.rs:55:60
   9: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
  10: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
  11: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
  12: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  13: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  14: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  15: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  16: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  17: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  18: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:63:9
  19: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  20: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  21: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  22: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  23: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  24: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  25: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  26: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  27: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  28: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  29: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  30: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  31: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  32: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  33: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  34: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  35: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  36: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  37: __libc_start_main
  38: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  34: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  35: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  36: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  37: __libc_start_main
  38: _start


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-13132-1715300067-01b24e7e3548382dd25","timestamp":"2024-05-10T00:20:45.723777Z","message":"Deleting namespace forge-framework-upgrade-pr-13132: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-framework-upgrade-pr-13132-1715300067-01b24e7e3548382dd25","timestamp":"2024-05-10T00:20:45.723799Z","message":"aptos-node resources for Forge removed in namespace: forge-framework-upgrade-pr-13132"}

failures:
    framework_upgrade::framework-upgrade

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::msg
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:83:36
   1: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:618:13
   2: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   3: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
   4: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
   5: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
   6: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
   7: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
   8: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
   9: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  10: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  11: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  12: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  13: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  14: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  15: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  16: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  17: __libc_start_main
  18: _start
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-validator-0                0/1     Pending     0          5m4s
aptos-node-1-validator-0                1/1     Running     0          6m5s
aptos-node-2-validator-0                1/1     Running     0          6m5s
aptos-node-3-validator-0                1/1     Running     0          6m5s
genesis-aptos-genesis-eforge131-7wnjw   0/1     Completed   0          16m
genesis-aptos-genesis-eforge229-kvd7l   0/1     Completed   0          41m
genesis-aptos-genesis-eforge33-29dkj    0/1     Completed   0          6m17s

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite framework_upgrade failure on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077 (PR)
Upgrade the nodes to version: 2466418b71d3b1a750390feef760bb56962f6077
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1321.9194263358963 txn/s, submitted: 1324.1016117425404 txn/s, failed submission: 2.1821854066439963 txn/s, expired: 2.1821854066439963 txn/s, latency: 2478.810986793837 ms, (p50: 2100 ms, p90: 4200 ms, p99: 6600 ms), latency samples: 109040
Test Failed: API error: Unknown error error sending request for url (http://aptos-node-3-validator.forge-framework-upgrade-pr-13132.svc:8080/v1/accounts/0000000000000000000000000000000000000000000000000000000000000001/resource/0x1::block::BlockResource): error trying to connect: dns error: failed to lookup address information: Name or service not known

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_release_builder::validate::NetworkConfig::increase_lockup::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:383:9
   3: aptos_release_builder::validate::execute_release::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:404:38
   4: aptos_release_builder::validate::validate_config_and_generate_release::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:493:6
   5: aptos_release_builder::validate::validate_config::{{closure}}
             at ./aptos-move/aptos-release-builder/src/validate.rs:479:80
   6: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   7: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
   8: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
   9: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
  10: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
  11: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  12: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  13: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  14: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  15: <aptos_testcases::framework_upgrade::FrameworkUpgrade as aptos_forge::interface::network::NetworkTest>::run
             at ./testsuite/testcases/src/framework_upgrade.rs:112:9
  16: aptos_forge::runner::Forge<F>::run::{{closure}}
             at ./testsuite/forge/src/runner.rs:598:42
  17: aptos_forge::runner::run_test
             at ./testsuite/forge/src/runner.rs:666:11
  18: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:598:30
  19: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  20: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  21: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  22: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  23: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  24: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  25: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  26: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  27: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  28: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  29: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  30: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  32: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  33: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  34: __libc_start_main
  35: _start
Trailing Log Lines:
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  31: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  32: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  33: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  34: __libc_start_main
  35: _start


Swarm logs can be found here: See fgi output for more information.
thread 'main' panicked at testsuite/forge/src/backend/k8s/swarm.rs:696:18:
called `Result::unwrap()` on an `Err` value: ApiError: namespaces "forge-framework-upgrade-pr-13132" not found: NotFound (ErrorResponse { status: "Failure", message: "namespaces \"forge-framework-upgrade-pr-13132\" not found", reason: "NotFound", code: 404 })

Caused by:
    namespaces "forge-framework-upgrade-pr-13132" not found: NotFound

Stack backtrace:
   0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/anyhow-1.0.79/src/error.rs:565:25
   1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1963:27
   2: aptos_forge::backend::k8s::cluster_helper::delete_k8s_cluster::{{closure}}
             at ./testsuite/forge/src/backend/k8s/cluster_helper.rs:289:13
   3: aptos_forge::backend::k8s::cluster_helper::uninstall_testnet_resources::{{closure}}
             at ./testsuite/forge/src/backend/k8s/cluster_helper.rs:399:48
   4: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
   5: tokio::runtime::coop::with_budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
   6: tokio::runtime::coop::budget
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
   7: tokio::runtime::park::CachedParkThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
   8: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
   9: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
  10: tokio::runtime::context::runtime::enter_runtime
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
  11: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
  12: tokio::runtime::runtime::Runtime::block_on
             at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:50
  13: <aptos_forge::backend::k8s::swarm::K8sSwarm as core::ops::drop::Drop>::drop
             at ./testsuite/forge/src/backend/k8s/swarm.rs:694:13
  14: core::ptr::drop_in_place<aptos_forge::backend::k8s::swarm::K8sSwarm>
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ptr/mod.rs:498:1
  15: core::ptr::drop_in_place<alloc::boxed::Box<dyn aptos_forge::interface::swarm::Swarm>>
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ptr/mod.rs:498:1
  16: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:611:9
  17: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
  18: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  19: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
  20: std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
  21: std::rt::lang_start::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
  22: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
  23: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  24: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  25: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  26: std::rt::lang_start_internal::{{closure}}
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
  27: std::panicking::try::do_call
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
  28: std::panicking::try
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
  29: std::panic::catch_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
  30: std::rt::lang_start_internal
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
  31: std::rt::lang_start
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
  32: __libc_start_main
  33: _start
stack backtrace:
   0: rust_begin_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
   2: core::result::unwrap_failed
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1653:5
   3: core::result::Result<T,E>::unwrap
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:1077:23
   4: <aptos_forge::backend::k8s::swarm::K8sSwarm as core::ops::drop::Drop>::drop
             at ./testsuite/forge/src/backend/k8s/swarm.rs:694:13
   5: core::ptr::drop_in_place<aptos_forge::backend::k8s::swarm::K8sSwarm>
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ptr/mod.rs:498:1
   6: core::ptr::drop_in_place<alloc::boxed::Box<dyn aptos_forge::interface::swarm::Swarm>>
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ptr/mod.rs:498:1
   7: aptos_forge::runner::Forge<F>::run
             at ./testsuite/forge/src/runner.rs:611:9
   8: forge::run_forge
             at ./testsuite/forge-cli/src/main.rs:427:11
   9: forge::main
             at ./testsuite/forge-cli/src/main.rs:353:21
  10: core::ops::function::FnOnce::call_once
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Debugging output:

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite framework_upgrade on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

• Grafana dashboard (auto-refresh)
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite framework_upgrade success on 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077

Compatibility test results for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077 (PR)
Upgrade the nodes to version: 2466418b71d3b1a750390feef760bb56962f6077
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1225.7520566638532 txn/s, submitted: 1228.6108253643608 txn/s, failed submission: 2.858768700507731 txn/s, expired: 2.858768700507731 txn/s, latency: 2410.239307499103 ms, (p50: 2100 ms, p90: 3900 ms, p99: 5900 ms), latency samples: 111480
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1106.169039406265 txn/s, submitted: 1109.2515168249927 txn/s, failed submission: 3.08247741872765 txn/s, expired: 3.08247741872765 txn/s, latency: 2646.7698049363057 ms, (p50: 2100 ms, p90: 4200 ms, p99: 7300 ms), latency samples: 100480
5. check swarm health
Compatibility test for 01b24e7e3548382dd25440b39a0438a993387f12 ==> 2466418b71d3b1a750390feef760bb56962f6077 passed
Upgrade the remaining nodes to version: 2466418b71d3b1a750390feef760bb56962f6077
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 583.8882918910747 txn/s, submitted: 585.239885159341 txn/s, failed submission: 1.3515932682663765 txn/s, expired: 1.3515932682663765 txn/s, latency: 5171.498456790124 ms, (p50: 2700 ms, p90: 12900 ms, p99: 14800 ms), latency samples: 51840
Test Ok

• Grafana dashboard
• Humio Logs
• Axiom Logs
• Validator CPU Profile
• Fullnode CPU Profile
• Test runner output
• Test run is land-blocking