aptos-core
aptos-core copied to clipboard

Published 20 hours ago •

Reame
Issues

reconfigure with dkg

Open zjma opened this issue 1 year ago • 24 comments

trafficstars

Description

The new reconfiguration process required by DKG, where:

Regular on-chain config changes are held in a buffer
When the current epoch expires:
1. disable txns that update on-chain configs;
2. perform a DKG;
3. apply the buffered on-chain config changes
4. enable txns that update on-chain configs
5. enter the new epoch using the default reconfigure().

Dec 21 '23 01:12 zjma

⏱️ 26h 25m total CI duration on this PR

Job	Cumulative Duration	Recent Runs
rust-unit-tests	4h 7m	🟥 🟥 🟥 🟥 🟥 (+7 more)
forge-framework-upgrade-test / forge	3h 54m	🟥 ⬜ ⬜ 🟥 🟥 (+3 more)
windows-build	2h 54m	🟩 🟩 🟩 🟩 🟩 (+7 more)
execution-performance / single-node-performance	2h 39m	🟩 🟩 🟩 🟩 🟩 (+3 more)
forge-e2e-test / forge	1h 47m	🟥 ⬜ ⬜ 🟩 🟩 (+3 more)
forge-compat-test / forge	1h 42m	🟩 ⬜ ⬜ 🟩 🟩 (+3 more)
rust-images / rust-all	1h 31m	🟩 🟩 🟩 🟩 🟩 (+4 more)
rust-smoke-tests	1h 26m	🟥 🟥 🟥 🟥 🟥 (+2 more)
rust-lints	1h 24m	🟩 🟩 🟩 🟩 🟩 (+7 more)
cli-e2e-tests / run-cli-tests	1h 13m	🟩 🟩 🟩 🟩 🟩 (+3 more)
run-tests-main-branch	59m	🟩 🟩 🟩 🟩 🟩 (+9 more)
check	49m	🟩 🟩 🟩 🟩 🟩 (+7 more)
general-lints	32m	🟩 🟩 🟩 🟩 🟩 (+7 more)
check-dynamic-deps	27m	🟩 🟩 🟩 🟩 🟩 (+7 more)
rust-unit-coverage	14m	🟥
indexer-grpc-e2e-tests / test-indexer-grpc-docker-compose	13m	🟥 🟥 🟥 🟩 🟩 (+3 more)
rust-smoke-coverage	10m	🟥
node-api-compatibility-tests / node-api-compatibility-tests	7m	🟩 🟩 🟩 🟩 🟩 (+3 more)
semgrep/ci	5m	🟩 🟩 🟩 🟩 🟩 (+7 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+9 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+7 more)
file_change_determinator	2m	🟩 🟩 🟩 🟩 🟩 (+4 more)
execution-performance / file_change_determinator	1m	🟩 🟩 🟩 🟩 🟩 (+2 more)
execution-performance / sequential-execution-performance	56s	🟩 🟩 🟩 🟩 🟩 (+2 more)
execution-performance / parallel-execution-performance	52s	🟩 🟩 🟩 🟩 🟩 (+2 more)
permission-check	48s	🟩 🟩 🟩 🟩 🟩 (+9 more)
permission-check	45s	🟩 🟩 🟩 🟩 🟩 (+9 more)
permission-check	41s	🟩 🟩 🟩 🟩 🟩 (+9 more)
permission-check	36s	🟩 🟩 🟩 🟩 🟩 (+9 more)
permission-check	26s	🟩 🟩 🟩 🟩 🟩 (+4 more)
determine-docker-build-metadata	20s	🟩 🟩 🟩 🟩 🟩 (+4 more)

🚨 3 jobs on the last run were significantly faster/slower than expected

Job	Duration	vs 7d avg	Delta
check-dynamic-deps	3m	2m
rust-lints	12m	8m
cli-e2e-tests / run-cli-tests	10m	8m

_{settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io}

Dec 21 '23 01:12 trunk-io[bot]

Forge is running suite `framework_upgrade` on `aptos-node-v1.8.3` ==> `e165eb0d730c19453ace59e1ea941b2f9ae1247e`

Dec 21 '23 04:12 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `e165eb0d730c19453ace59e1ea941b2f9ae1247e`

Dec 21 '23 04:12 github-actions[bot]

Forge is running suite `compat` on `aptos-node-v1.8.3` ==> `e165eb0d730c19453ace59e1ea941b2f9ae1247e`

Dec 21 '23 04:12 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `e165eb0d730c19453ace59e1ea941b2f9ae1247e`

two traffics test: inner traffic : committed: 8937 txn/s, submitted: 8938 txn/s, latency: 4375 ms, (p50: 4200 ms, p90: 4800 ms, p99: 9900 ms), latency samples: 3851868
two traffics test : committed: 100 txn/s, latency: 2183 ms, (p50: 2100 ms, p90: 2500 ms, p99: 4000 ms), latency samples: 1760
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.205, avg: 0.197", "QsPosToProposal: max: 0.132, avg: 0.125", "ConsensusProposalToOrdered: max: 0.526, avg: 0.507", "ConsensusOrderedToCommit: max: 0.498, avg: 0.475", "ConsensusProposalToCommit: max: 1.006, avg: 0.981"]
Max round gap was 1 [limit 4] at version 1950178. Max no progress secs was 4.677179 [limit 10] at version 1950178.
Test Ok

Dec 21 '23 04:12 github-actions[bot]

:white_check_mark: Forge suite `compat` success on `aptos-node-v1.8.3` ==> `e165eb0d730c19453ace59e1ea941b2f9ae1247e`

Compatibility test results for aptos-node-v1.8.3 ==> e165eb0d730c19453ace59e1ea941b2f9ae1247e (PR)
1. Check liveness of validators at old version: aptos-node-v1.8.3
compatibility::simple-validator-upgrade::liveness-check : committed: 4967 txn/s, latency: 6372 ms, (p50: 6300 ms, p90: 8400 ms, p99: 16200 ms), latency samples: 198700
2. Upgrading first Validator to new version: e165eb0d730c19453ace59e1ea941b2f9ae1247e
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1878 txn/s, latency: 15498 ms, (p50: 19000 ms, p90: 21900 ms, p99: 22200 ms), latency samples: 92040
3. Upgrading rest of first batch to new version: e165eb0d730c19453ace59e1ea941b2f9ae1247e
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1583 txn/s, latency: 17494 ms, (p50: 18600 ms, p90: 23400 ms, p99: 28600 ms), latency samples: 80760
4. upgrading second batch to new version: e165eb0d730c19453ace59e1ea941b2f9ae1247e
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3076 txn/s, latency: 10259 ms, (p50: 9600 ms, p90: 18900 ms, p99: 19600 ms), latency samples: 123040
5. check swarm health
Compatibility test for aptos-node-v1.8.3 ==> e165eb0d730c19453ace59e1ea941b2f9ae1247e passed
Test Ok

Dec 21 '23 04:12 github-actions[bot]

Forge is running suite `compat` on `aptos-node-v1.8.3` ==> `d00d9439868ab4db66be78d2643a2153e2df1948`

Dec 21 '23 04:12 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `d00d9439868ab4db66be78d2643a2153e2df1948`

Dec 21 '23 04:12 github-actions[bot]

Forge is running suite `framework_upgrade` on `aptos-node-v1.8.3` ==> `d00d9439868ab4db66be78d2643a2153e2df1948`

Dec 21 '23 04:12 github-actions[bot]

:white_check_mark: Forge suite `compat` success on `aptos-node-v1.8.3` ==> `d00d9439868ab4db66be78d2643a2153e2df1948`

Compatibility test results for aptos-node-v1.8.3 ==> d00d9439868ab4db66be78d2643a2153e2df1948 (PR)
1. Check liveness of validators at old version: aptos-node-v1.8.3
compatibility::simple-validator-upgrade::liveness-check : committed: 3176 txn/s, latency: 6908 ms, (p50: 6100 ms, p90: 10200 ms, p99: 16700 ms), latency samples: 174720
2. Upgrading first Validator to new version: d00d9439868ab4db66be78d2643a2153e2df1948
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1864 txn/s, latency: 15849 ms, (p50: 18600 ms, p90: 22000 ms, p99: 22300 ms), latency samples: 93220
3. Upgrading rest of first batch to new version: d00d9439868ab4db66be78d2643a2153e2df1948
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1770 txn/s, latency: 16562 ms, (p50: 19000 ms, p90: 22200 ms, p99: 22300 ms), latency samples: 92040
4. upgrading second batch to new version: d00d9439868ab4db66be78d2643a2153e2df1948
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3455 txn/s, latency: 9160 ms, (p50: 8700 ms, p90: 13500 ms, p99: 13800 ms), latency samples: 138200
5. check swarm health
Compatibility test for aptos-node-v1.8.3 ==> d00d9439868ab4db66be78d2643a2153e2df1948 passed
Test Ok

Dec 21 '23 04:12 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `d00d9439868ab4db66be78d2643a2153e2df1948`

two traffics test: inner traffic : committed: 7341 txn/s, submitted: 7348 txn/s, expired: 6 txn/s, latency: 4718 ms, (p50: 4500 ms, p90: 5700 ms, p99: 10800 ms), latency samples: 3516588
two traffics test : committed: 100 txn/s, latency: 2238 ms, (p50: 2200 ms, p90: 2700 ms, p99: 4100 ms), latency samples: 1960
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.227, avg: 0.199", "QsPosToProposal: max: 0.150, avg: 0.142", "ConsensusProposalToOrdered: max: 0.578, avg: 0.550", "ConsensusOrderedToCommit: max: 0.499, avg: 0.475", "ConsensusProposalToCommit: max: 1.050, avg: 1.025"]
Max round gap was 1 [limit 4] at version 1738277. Max no progress secs was 5.932069 [limit 10] at version 1738277.
Test Ok

Dec 21 '23 04:12 github-actions[bot]

:x: Forge suite `framework_upgrade` failure on `aptos-node-v1.8.3` ==> `d00d9439868ab4db66be78d2643a2153e2df1948`

Compatibility test results for aptos-node-v1.8.3 ==> d00d9439868ab4db66be78d2643a2153e2df1948 (PR)
Upgrade the nodes to version: d00d9439868ab4db66be78d2643a2153e2df1948
Test Failed: API error: Unknown error error sending request for url (http://aptos-node-3-validator.forge-framework-upgrade-pr-11458.svc:8080/v1/estimate_gas_price): error trying to connect: dns error: failed to lookup address information: Name or service not known

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: __libc_start_main
  13: <unknown>
Trailing Log Lines:
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: __libc_start_main
  13: <unknown>


Swarm logs can be found here: See fgi output for more information.
thread 'main' panicked at testsuite/forge/src/backend/k8s/swarm.rs:676:18:
called `Result::unwrap()` on an `Err` value: ApiError: namespaces "forge-framework-upgrade-pr-11458" not found: NotFound (ErrorResponse { status: "Failure", message: "namespaces \"forge-framework-upgrade-pr-11458\" not found", reason: "NotFound", code: 404 })

Caused by:
    namespaces "forge-framework-upgrade-pr-11458" not found: NotFound

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: __libc_start_main
  15: <unknown>
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Debugging output:

Dec 21 '23 05:12 github-actions[bot]

Forge is running suite `compat` on `aptos-node-v1.8.3` ==> `22b9836456350a1f93df41d0d8623ef680b509ab`

Dec 22 '23 03:12 github-actions[bot]

Forge is running suite `framework_upgrade` on `aptos-node-v1.8.3` ==> `22b9836456350a1f93df41d0d8623ef680b509ab`

Dec 22 '23 03:12 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `22b9836456350a1f93df41d0d8623ef680b509ab`

Dec 22 '23 03:12 github-actions[bot]

:white_check_mark: Forge suite `compat` success on `aptos-node-v1.8.3` ==> `22b9836456350a1f93df41d0d8623ef680b509ab`

Compatibility test results for aptos-node-v1.8.3 ==> 22b9836456350a1f93df41d0d8623ef680b509ab (PR)
1. Check liveness of validators at old version: aptos-node-v1.8.3
compatibility::simple-validator-upgrade::liveness-check : committed: 4976 txn/s, latency: 6513 ms, (p50: 6800 ms, p90: 9600 ms, p99: 10500 ms), latency samples: 189120
2. Upgrading first Validator to new version: 22b9836456350a1f93df41d0d8623ef680b509ab
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1780 txn/s, latency: 16008 ms, (p50: 19000 ms, p90: 22000 ms, p99: 22500 ms), latency samples: 92600
3. Upgrading rest of first batch to new version: 22b9836456350a1f93df41d0d8623ef680b509ab
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1477 txn/s, latency: 18757 ms, (p50: 20300 ms, p90: 27200 ms, p99: 30800 ms), latency samples: 70920
4. upgrading second batch to new version: 22b9836456350a1f93df41d0d8623ef680b509ab
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 3220 txn/s, latency: 8779 ms, (p50: 9300 ms, p90: 11600 ms, p99: 25200 ms), latency samples: 141680
5. check swarm health
Compatibility test for aptos-node-v1.8.3 ==> 22b9836456350a1f93df41d0d8623ef680b509ab passed
Test Ok

Dec 22 '23 03:12 github-actions[bot]

:white_check_mark: Forge suite `realistic_env_max_load` success on `22b9836456350a1f93df41d0d8623ef680b509ab`

two traffics test: inner traffic : committed: 7056 txn/s, submitted: 7059 txn/s, expired: 3 txn/s, latency: 4916 ms, (p50: 4600 ms, p90: 6000 ms, p99: 10100 ms), latency samples: 3394129
two traffics test : committed: 100 txn/s, latency: 2206 ms, (p50: 2200 ms, p90: 2600 ms, p99: 3100 ms), latency samples: 2040
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.213, avg: 0.196", "QsPosToProposal: max: 0.167, avg: 0.153", "ConsensusProposalToOrdered: max: 0.599, avg: 0.560", "ConsensusOrderedToCommit: max: 0.558, avg: 0.529", "ConsensusProposalToCommit: max: 1.128, avg: 1.089"]
Max round gap was 1 [limit 4] at version 1530115. Max no progress secs was 3.993486 [limit 10] at version 1530115.
Test Ok

Dec 22 '23 03:12 github-actions[bot]

:x: Forge suite `framework_upgrade` failure on `aptos-node-v1.8.3` ==> `22b9836456350a1f93df41d0d8623ef680b509ab`

Compatibility test results for aptos-node-v1.8.3 ==> 22b9836456350a1f93df41d0d8623ef680b509ab (PR)
Upgrade the nodes to version: 22b9836456350a1f93df41d0d8623ef680b509ab
Test Failed: API error: Unknown error error sending request for url (http://aptos-node-3-validator.forge-framework-upgrade-pr-11458.svc:8080/v1/estimate_gas_price): error trying to connect: dns error: failed to lookup address information: Name or service not known

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: __libc_start_main
  14: <unknown>
Trailing Log Lines:
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: __libc_start_main
  14: <unknown>


Swarm logs can be found here: See fgi output for more information.
thread 'main' panicked at testsuite/forge/src/backend/k8s/swarm.rs:676:18:
called `Result::unwrap()` on an `Err` value: ApiError: namespaces "forge-framework-upgrade-pr-11458" not found: NotFound (ErrorResponse { status: "Failure", message: "namespaces \"forge-framework-upgrade-pr-11458\" not found", reason: "NotFound", code: 404 })

Caused by:
    namespaces "forge-framework-upgrade-pr-11458" not found: NotFound

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: __libc_start_main
  15: <unknown>
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Debugging output:

Dec 22 '23 03:12 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `4eba00a23c225f9b74f1f40734e1dc753def3205`

Dec 24 '23 05:12 github-actions[bot]

Forge is running suite `framework_upgrade` on `aptos-node-v1.8.3` ==> `4eba00a23c225f9b74f1f40734e1dc753def3205`

Dec 24 '23 05:12 github-actions[bot]

Forge is running suite `compat` on `aptos-node-v1.8.3` ==> `4eba00a23c225f9b74f1f40734e1dc753def3205`

Dec 24 '23 05:12 github-actions[bot]

:x: Forge suite `realistic_env_max_load` failure on `4eba00a23c225f9b74f1f40734e1dc753def3205`

two traffics test: inner traffic : committed: 6769 txn/s, submitted: 6775 txn/s, expired: 5 txn/s, latency: 5035 ms, (p50: 4800 ms, p90: 6300 ms, p99: 9800 ms), latency samples: 3296886
Test Failed: test NetworkLoadTest

Caused by:
    TPS requirement for inner traffic failed. Average TPS 6769, minimum TPS requirement 7000. Full stats: committed: 6769 txn/s, submitted: 6775 txn/s, expired: 5 txn/s, latency: 5035 ms, (p50: 4800 ms, p90: 6300 ms, p99: 9800 ms), latency samples: 3296886

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: __libc_start_main
  13: <unknown>
Trailing Log Lines:
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: __libc_start_main
  13: <unknown>


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-11458-1703396497-4eba00a23c225f9b74f1f40734e1dc753","timestamp":"2023-12-24T05:53:17.358134Z","message":"Deleting namespace forge-e2e-pr-11458: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-11458-1703396497-4eba00a23c225f9b74f1f40734e1dc753","timestamp":"2023-12-24T05:53:17.358171Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-11458"}
Failed to run tests:
Tests Failed

failures:
    CompositeNetworkTest

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Error: Tests Failed

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: __libc_start_main
   6: <unknown>
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge254-0       1/1     Running     0          11m
aptos-node-0-validator-0                1/1     Running     0          11m
aptos-node-1-fullnode-eforge254-0       1/1     Running     0          11m
aptos-node-1-validator-0                1/1     Running     0          11m
aptos-node-2-fullnode-eforge254-0       1/1     Running     0          11m
aptos-node-2-validator-0                1/1     Running     0          11m
aptos-node-3-fullnode-eforge254-0       1/1     Running     0          11m
aptos-node-3-validator-0                1/1     Running     0          11m
aptos-node-4-fullnode-eforge254-0       1/1     Running     0          11m
aptos-node-4-validator-0                1/1     Running     0          11m
aptos-node-5-validator-0                1/1     Running     0          11m
aptos-node-6-validator-0                1/1     Running     0          11m
genesis-aptos-genesis-eforge254-bhlpk   0/1     Completed   0          11m

Dec 24 '23 05:12 github-actions[bot]

:white_check_mark: Forge suite `compat` success on `aptos-node-v1.8.3` ==> `4eba00a23c225f9b74f1f40734e1dc753def3205`

Compatibility test results for aptos-node-v1.8.3 ==> 4eba00a23c225f9b74f1f40734e1dc753def3205 (PR)
1. Check liveness of validators at old version: aptos-node-v1.8.3
compatibility::simple-validator-upgrade::liveness-check : committed: 3671 txn/s, latency: 6624 ms, (p50: 6900 ms, p90: 9700 ms, p99: 12300 ms), latency samples: 187260
2. Upgrading first Validator to new version: 4eba00a23c225f9b74f1f40734e1dc753def3205
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1845 txn/s, latency: 15620 ms, (p50: 18400 ms, p90: 22100 ms, p99: 22600 ms), latency samples: 92280
3. Upgrading rest of first batch to new version: 4eba00a23c225f9b74f1f40734e1dc753def3205
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 1768 txn/s, latency: 16053 ms, (p50: 19000 ms, p90: 22200 ms, p99: 22600 ms), latency samples: 91960
4. upgrading second batch to new version: 4eba00a23c225f9b74f1f40734e1dc753def3205
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 2409 txn/s, latency: 8987 ms, (p50: 9600 ms, p90: 12700 ms, p99: 13000 ms), latency samples: 144560
5. check swarm health
Compatibility test for aptos-node-v1.8.3 ==> 4eba00a23c225f9b74f1f40734e1dc753def3205 passed
Test Ok

Dec 24 '23 05:12 github-actions[bot]

:x: Forge suite `framework_upgrade` failure on `aptos-node-v1.8.3` ==> `4eba00a23c225f9b74f1f40734e1dc753def3205`

Compatibility test results for aptos-node-v1.8.3 ==> 4eba00a23c225f9b74f1f40734e1dc753def3205 (PR)
Upgrade the nodes to version: 4eba00a23c225f9b74f1f40734e1dc753def3205
Test Failed: API error: Unknown error error sending request for url (http://aptos-node-3-validator.forge-framework-upgrade-pr-11458.svc:8080/v1/estimate_gas_price): error trying to connect: dns error: failed to lookup address information: Name or service not known

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: __libc_start_main
  14: <unknown>
Trailing Log Lines:
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: __libc_start_main
  14: <unknown>


Swarm logs can be found here: See fgi output for more information.
thread 'main' panicked at testsuite/forge/src/backend/k8s/swarm.rs:676:18:
called `Result::unwrap()` on an `Err` value: ApiError: namespaces "forge-framework-upgrade-pr-11458" not found: NotFound (ErrorResponse { status: "Failure", message: "namespaces \"forge-framework-upgrade-pr-11458\" not found", reason: "NotFound", code: 404 })

Caused by:
    namespaces "forge-framework-upgrade-pr-11458" not found: NotFound

Stack backtrace:
   0: <unknown>
   1: <unknown>
   2: <unknown>
   3: <unknown>
   4: <unknown>
   5: <unknown>
   6: <unknown>
   7: <unknown>
   8: <unknown>
   9: <unknown>
  10: <unknown>
  11: <unknown>
  12: <unknown>
  13: <unknown>
  14: __libc_start_main
  15: <unknown>
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Debugging output:

Dec 24 '23 06:12 github-actions[bot]

Forge is running suite `compat` on `aptos-node-v1.8.3` ==> `4c7d06d1a744be7e5408d2cdc0d89438132f5b9f`

Jan 08 '24 03:01 github-actions[bot]

Forge is running suite `framework_upgrade` on `aptos-node-v1.8.3` ==> `4c7d06d1a744be7e5408d2cdc0d89438132f5b9f`

Jan 08 '24 03:01 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `4c7d06d1a744be7e5408d2cdc0d89438132f5b9f`

Jan 08 '24 03:01 github-actions[bot]

Forge is running suite `realistic_env_max_load` on `e20e5f70ac51907adf5708ae08f18b6acac7a6c8`

Jan 08 '24 04:01 github-actions[bot]

Forge is running suite `compat` on `aptos-node-v1.8.3` ==> `e20e5f70ac51907adf5708ae08f18b6acac7a6c8`

Jan 08 '24 04:01 github-actions[bot]

Forge is running suite `framework_upgrade` on `aptos-node-v1.8.3` ==> `e20e5f70ac51907adf5708ae08f18b6acac7a6c8`

Jan 08 '24 04:01 github-actions[bot]