aptos-core
aptos-core copied to clipboard
Published
20 hours ago •
aptos-labs
aptos-labs
[DRAFT] Quorum store reset
Description
Revert to earlier commit for debugging purpose.
Test Plan
This change is 
Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 34c65c6648dbd2a63b0f706e3ae55b8f82e88c32
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Forge is running suite land_blocking on 34c65c6648dbd2a63b0f706e3ae55b8f82e88c32
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 34c65c6648dbd2a63b0f706e3ae55b8f82e88c32
Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 34c65c6648dbd2a63b0f706e3ae55b8f82e88c32 (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7523 TPS, 4956 ms latency, 7300 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 34c65c6648dbd2a63b0f706e3ae55b8f82e88c32
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x279933239ddd8faa8b292e4c7ae5c0697ddfa7084294a4987ccd8e744501333b): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Trailing Log Lines:
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7523 TPS, 4956 ms latency, 7300 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 34c65c6648dbd2a63b0f706e3ae55b8f82e88c32
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x279933239ddd8faa8b292e4c7ae5c0697ddfa7084294a4987ccd8e744501333b): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664578633-843b204dce971d98449b82624f4f684","timestamp":"2022-09-30T23:06:41.547792Z","message":"Deleting namespace forge-compat-pr-4687: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664578633-843b204dce971d98449b82624f4f684","timestamp":"2022-09-30T23:06:41.547820Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4687"}
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
:white_check_mark: Forge suite land_blocking success on 34c65c6648dbd2a63b0f706e3ae55b8f82e88c32
performance benchmark with full nodes : 7026 TPS, 3486 ms latency, 8800 ms p99 latency,(!) expired 149960 out of 7654520 txns
Test Ok
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 241fa149dd2e8abb747894efa7d9d3868f96d34f
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite land_blocking on 241fa149dd2e8abb747894efa7d9d3868f96d34f
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite land_blocking failure on 241fa149dd2e8abb747894efa7d9d3868f96d34f
Test Failed: start emitter job
Caused by:
Failed to wait for transactions: Unknown(Transaction expired, it is guaranteed it will not be committed on chain.)
Trailing Log Lines:
Error: start emitter job
Caused by:
Failed to wait for transactions: Unknown(Transaction expired, it is guaranteed it will not be committed on chain.)
Test Statistics:
Test Failed: start emitter job
Caused by:
Failed to wait for transactions: Unknown(Transaction expired, it is guaranteed it will not be committed on chain.)
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4687-1664591278-241fa149dd2e8abb747894efa7d9d3868f","timestamp":"2022-10-01T02:39:09.465742Z","message":"Deleting namespace forge-e2e-pr-4687: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4687-1664591278-241fa149dd2e8abb747894efa7d9d3868f","timestamp":"2022-10-01T02:39:09.465775Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4687"}
failures:
performance benchmark with full nodes
Failed to run tests:
Tests Failed
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-fullnode-eforge185-0 1/1 Running 0 4m56s
aptos-node-0-validator-0 1/1 Running 0 4m55s
aptos-node-1-fullnode-eforge185-0 1/1 Running 0 4m56s
aptos-node-1-validator-0 1/1 Running 0 4m55s
aptos-node-10-validator-0 1/1 Running 0 4m56s
aptos-node-11-validator-0 1/1 Running 0 4m55s
aptos-node-12-validator-0 1/1 Running 0 4m56s
aptos-node-13-validator-0 1/1 Running 0 4m56s
aptos-node-14-validator-0 1/1 Running 0 4m55s
aptos-node-15-validator-0 1/1 Running 0 4m55s
aptos-node-16-validator-0 1/1 Running 0 4m55s
aptos-node-17-validator-0 1/1 Running 0 4m55s
aptos-node-18-validator-0 1/1 Running 0 4m54s
aptos-node-19-validator-0 1/1 Running 0 4m56s
aptos-node-2-fullnode-eforge185-0 1/1 Running 0 4m56s
aptos-node-2-validator-0 1/1 Running 0 4m56s
aptos-node-3-fullnode-eforge185-0 1/1 Running 0 4m56s
aptos-node-3-validator-0 1/1 Running 0 4m55s
aptos-node-4-fullnode-eforge185-0 1/1 Running 0 4m56s
aptos-node-4-validator-0 1/1 Running 0 4m56s
aptos-node-5-fullnode-eforge185-0 1/1 Running 0 4m56s
aptos-node-5-validator-0 1/1 Running 0 4m56s
aptos-node-6-fullnode-eforge185-0 1/1 Running 0 4m55s
aptos-node-6-validator-0 1/1 Running 0 4m56s
aptos-node-7-fullnode-eforge185-0 1/1 Running 0 4m56s
aptos-node-7-validator-0 1/1 Running 0 4m55s
aptos-node-8-fullnode-eforge185-0 1/1 Running 0 4m54s
aptos-node-8-validator-0 1/1 Running 0 4m55s
aptos-node-9-fullnode-eforge185-0 1/1 Running 0 4m55s
aptos-node-9-validator-0 1/1 Running 1 (39s ago) 4m55s
genesis-aptos-genesis-eforge185-gs7bt 0/1 Completed 0 10m
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 241fa149dd2e8abb747894efa7d9d3868f96d34f
Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 241fa149dd2e8abb747894efa7d9d3868f96d34f (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7738 TPS, 4779 ms latency, 7800 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 241fa149dd2e8abb747894efa7d9d3868f96d34f
Test Failed: CrashLoopBackOff
Trailing Log Lines:
::error::CrashLoopBackOff
test compatibility::simple-validator-upgrade ... FAILED
Error: CrashLoopBackOff
Test Statistics:
Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 241fa149dd2e8abb747894efa7d9d3868f96d34f (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7738 TPS, 4779 ms latency, 7800 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 241fa149dd2e8abb747894efa7d9d3868f96d34f
Test Failed: CrashLoopBackOff
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664591274-843b204dce971d98449b82624f4f684","timestamp":"2022-10-01T02:40:51.017733Z","message":"Deleting namespace forge-compat-pr-4687: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664591274-843b204dce971d98449b82624f4f684","timestamp":"2022-10-01T02:40:51.017759Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4687"}
failures:
compatibility::simple-validator-upgrade
Failed to run tests:
Tests Failed
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 6m58s
aptos-node-1-validator-0 1/1 Running 2 (16s ago) 114s
aptos-node-2-validator-0 1/1 Running 0 6m58s
aptos-node-3-validator-0 1/1 Running 0 6m58s
aptos-node-4-validator-0 1/1 Running 0 6m58s
genesis-aptos-genesis-eforge70-kwqs5 0/1 Completed 0 12m
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite land_blocking on 816d140fde2819e28903d8124044dcfdfd7eff39
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 816d140fde2819e28903d8124044dcfdfd7eff39
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 816d140fde2819e28903d8124044dcfdfd7eff39
Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 816d140fde2819e28903d8124044dcfdfd7eff39 (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7338 TPS, 4903 ms latency, 7100 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 816d140fde2819e28903d8124044dcfdfd7eff39
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x1e8073d18eab63e58d2917a337b35674cb34abaa14478aa2bbb31fba1489d617): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Trailing Log Lines:
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7338 TPS, 4903 ms latency, 7100 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 816d140fde2819e28903d8124044dcfdfd7eff39
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x1e8073d18eab63e58d2917a337b35674cb34abaa14478aa2bbb31fba1489d617): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664737447-843b204dce971d98449b82624f4f684","timestamp":"2022-10-02T19:12:14.490073Z","message":"Deleting namespace forge-compat-pr-4687: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664737447-843b204dce971d98449b82624f4f684","timestamp":"2022-10-02T19:12:14.490104Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4687"}
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 4m53s
aptos-node-1-validator-0 0/1 CrashLoopBackOff 1 (11s ago) 111s
aptos-node-2-validator-0 1/1 Running 0 4m53s
aptos-node-3-validator-0 1/1 Running 0 4m53s
aptos-node-4-validator-0 1/1 Running 0 4m53s
genesis-aptos-genesis-eforge34-5qj95 0/1 Completed 0 7m26s
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
:white_check_mark: Forge suite land_blocking success on 816d140fde2819e28903d8124044dcfdfd7eff39
performance benchmark with full nodes : 6322 TPS, 3509 ms latency, 8700 ms p99 latency,(!) expired 178927 out of 6931700 txns
Test Ok
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite land_blocking on 3f5d8e35286e55a58228aebb7251df74e49defbf
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 3f5d8e35286e55a58228aebb7251df74e49defbf
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 3f5d8e35286e55a58228aebb7251df74e49defbf
Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 3f5d8e35286e55a58228aebb7251df74e49defbf (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7648 TPS, 4773 ms latency, 7200 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 3f5d8e35286e55a58228aebb7251df74e49defbf
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x4836287c9b0858b12e262ee00a2efa429e4dfae5a3c9c5bd60dad8626ab083e3): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Trailing Log Lines:
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7648 TPS, 4773 ms latency, 7200 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 3f5d8e35286e55a58228aebb7251df74e49defbf
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x4836287c9b0858b12e262ee00a2efa429e4dfae5a3c9c5bd60dad8626ab083e3): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664743427-843b204dce971d98449b82624f4f684","timestamp":"2022-10-02T20:51:31.726662Z","message":"Deleting namespace forge-compat-pr-4687: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664743427-843b204dce971d98449b82624f4f684","timestamp":"2022-10-02T20:51:31.726693Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4687"}
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
:white_check_mark: Forge suite land_blocking success on 3f5d8e35286e55a58228aebb7251df74e49defbf
performance benchmark with full nodes : 6395 TPS, 3587 ms latency, 11800 ms p99 latency,(!) expired 237326 out of 7068000 txns
Test Ok
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 04b4f030b5fadaaa4cd0c120fd647e577d11e3bf
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite land_blocking on 04b4f030b5fadaaa4cd0c120fd647e577d11e3bf
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 04b4f030b5fadaaa4cd0c120fd647e577d11e3bf
Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 04b4f030b5fadaaa4cd0c120fd647e577d11e3bf (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 8348 TPS, 4301 ms latency, 6500 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 04b4f030b5fadaaa4cd0c120fd647e577d11e3bf
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x215e2b1ed0f1c1aa0d763d9819e99579711a59a625af3cbdaeb313db783fb648): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Trailing Log Lines:
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 8348 TPS, 4301 ms latency, 6500 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 04b4f030b5fadaaa4cd0c120fd647e577d11e3bf
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x215e2b1ed0f1c1aa0d763d9819e99579711a59a625af3cbdaeb313db783fb648): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664825922-843b204dce971d98449b82624f4f684","timestamp":"2022-10-03T19:43:21.334049Z","message":"Deleting namespace forge-compat-pr-4687: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664825922-843b204dce971d98449b82624f4f684","timestamp":"2022-10-03T19:43:21.334087Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4687"}
Failed to run tests:
Tests Failed
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Error: Tests Failed
Debugging output:
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
:white_check_mark: Forge suite land_blocking success on 04b4f030b5fadaaa4cd0c120fd647e577d11e3bf
performance benchmark with full nodes : 6225 TPS, 3673 ms latency, 12100 ms p99 latency,(!) expired 221789 out of 6870360 txns
Test Ok
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite land_blocking on 04b4f030b5fadaaa4cd0c120fd647e577d11e3bf
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite land_blocking failure on 04b4f030b5fadaaa4cd0c120fd647e577d11e3bf
performance benchmark with full nodes : 6187 TPS, 3520 ms latency, 11400 ms p99 latency,(!) expired 245843 out of 6854160 txns
Test Failed: check for success
Caused by:
0: Failed ensuring no validator restarted
1: Container validator in pod aptos-node-19-validator-0 restarted 1 times
Trailing Log Lines:
0: Failed ensuring no validator restarted
1: Container validator in pod aptos-node-19-validator-0 restarted 1 times
Test Statistics:
performance benchmark with full nodes : 6187 TPS, 3520 ms latency, 11400 ms p99 latency,(!) expired 245843 out of 6854160 txns
Test Failed: check for success
Caused by:
0: Failed ensuring no validator restarted
1: Container validator in pod aptos-node-19-validator-0 restarted 1 times
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4687-1664831692-04b4f030b5fadaaa4cd0c120fd647e577d","timestamp":"2022-10-03T21:44:23.821702Z","message":"Deleting namespace forge-e2e-pr-4687: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4687-1664831692-04b4f030b5fadaaa4cd0c120fd647e577d","timestamp":"2022-10-03T21:44:23.821741Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4687"}
failures:
performance benchmark with full nodes
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 088e3b92b35bed094cc9f8e1cdbe37d0dd2c91ff
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite land_blocking on 088e3b92b35bed094cc9f8e1cdbe37d0dd2c91ff
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 088e3b92b35bed094cc9f8e1cdbe37d0dd2c91ff
Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 088e3b92b35bed094cc9f8e1cdbe37d0dd2c91ff (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7732 TPS, 4957 ms latency, 7100 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 088e3b92b35bed094cc9f8e1cdbe37d0dd2c91ff
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x57ef9c9e824957ea30e30769ccad3ca4ac153e2f9e3a8333bc29b29f6eb23bed): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Trailing Log Lines:
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7732 TPS, 4957 ms latency, 7100 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 088e3b92b35bed094cc9f8e1cdbe37d0dd2c91ff
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x57ef9c9e824957ea30e30769ccad3ca4ac153e2f9e3a8333bc29b29f6eb23bed): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664840783-843b204dce971d98449b82624f4f684","timestamp":"2022-10-03T23:57:05.748174Z","message":"Deleting namespace forge-compat-pr-4687: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664840783-843b204dce971d98449b82624f4f684","timestamp":"2022-10-03T23:57:05.748203Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4687"}
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-validator-0 1/1 Running 0 7m47s
aptos-node-1-validator-0 0/1 CrashLoopBackOff 1 (8s ago) 102s
aptos-node-2-validator-0 1/1 Running 0 7m47s
aptos-node-3-validator-0 1/1 Running 0 7m47s
aptos-node-4-validator-0 1/1 Running 0 7m47s
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
:white_check_mark: Forge suite land_blocking success on 088e3b92b35bed094cc9f8e1cdbe37d0dd2c91ff
performance benchmark with full nodes : 6590 TPS, 3575 ms latency, 12000 ms p99 latency,(!) expired 225059 out of 7263220 txns
Test Ok
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 4cb3ed6004e39b8f71edbd457975ca7ae5486983
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
Forge is running suite land_blocking on 4cb3ed6004e39b8f71edbd457975ca7ae5486983
- Grafana dashboard (auto-refresh)
- Humio Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 4cb3ed6004e39b8f71edbd457975ca7ae5486983
Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 4cb3ed6004e39b8f71edbd457975ca7ae5486983 (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 8240 TPS, 4640 ms latency, 6800 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 4cb3ed6004e39b8f71edbd457975ca7ae5486983
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x5cc6b28c356ce8a0e176a4a15f8f2271f23933a591fa9d84e9955b69ab326462): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Trailing Log Lines:
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 8240 TPS, 4640 ms latency, 6800 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 4cb3ed6004e39b8f71edbd457975ca7ae5486983
Test Failed: Failed to wait for transactions: Unknown(error sending request for url (http://aptos-node-1-validator.forge-compat-pr-4687.svc:8080/v1/transactions/by_hash/0x5cc6b28c356ce8a0e176a4a15f8f2271f23933a591fa9d84e9955b69ab326462): error trying to connect: tcp connect error: Connection refused (os error 111)
Caused by:
0: error trying to connect: tcp connect error: Connection refused (os error 111)
1: tcp connect error: Connection refused (os error 111)
2: Connection refused (os error 111))
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664847398-843b204dce971d98449b82624f4f684","timestamp":"2022-10-04T01:46:01.505649Z","message":"Deleting namespace forge-compat-pr-4687: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-compat-pr-4687-1664847398-843b204dce971d98449b82624f4f684","timestamp":"2022-10-04T01:46:01.505682Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4687"}
failures:
compatibility::simple-validator-upgrade
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking
:white_check_mark: Forge suite land_blocking success on 4cb3ed6004e39b8f71edbd457975ca7ae5486983
performance benchmark with full nodes : 6539 TPS, 3796 ms latency, 13900 ms p99 latency,(!) expired 215323 out of 7199860 txns
Test Ok
- Grafana dashboard
- Humio Logs
- Test runner output
- Test run is land-blocking