aptos-core icon indicating copy to clipboard operation
aptos-core copied to clipboard

Published 20 hours ago • verified publisher icon aptos-labs

Twin validator test

Open zjma opened this issue 3 years ago • 1 comments

Description

Test Plan

This change is Reviewable

zjma avatar Sep 12 '22 20:09 zjma

Forge is running suite twin_validator_test on testnet ==> 69cfec5debe1541385b53554ea581c37d9735ed4

Forge is running suite twin_validator_test on 69cfec5debe1541385b53554ea581c37d9735ed4

:white_check_mark: Forge suite twin_validator_test success on testnet ==> 69cfec5debe1541385b53554ea581c37d9735ed4

twin validator : 3842 TPS, 7248 ms latency, 18900 ms p99 latency,no expired txns
Test Ok

:white_check_mark: Forge suite twin_validator_test success on 69cfec5debe1541385b53554ea581c37d9735ed4

twin validator : 3573 TPS, 7951 ms latency, 20500 ms p99 latency,no expired txns
Test Ok

Forge is running suite twin_validator_test on 91b7a6105ffb4198957a290831f636b8f7f8aa6a

Forge is running suite twin_validator_test on testnet ==> 91b7a6105ffb4198957a290831f636b8f7f8aa6a

:white_check_mark: Forge suite twin_validator_test success on testnet ==> 91b7a6105ffb4198957a290831f636b8f7f8aa6a

twin validator : 7714 TPS, 3854 ms latency, 17800 ms p99 latency,no expired txns
Test Ok

:white_check_mark: Forge suite twin_validator_test success on 91b7a6105ffb4198957a290831f636b8f7f8aa6a

twin validator : 7507 TPS, 3913 ms latency, 9900 ms p99 latency,no expired txns
Test Ok

Forge is running suite twin_validator_test on a7f95bdafc94c901373e5b3241bb99ed76f140dc

Forge is running suite twin_validator_test on testnet ==> a7f95bdafc94c901373e5b3241bb99ed76f140dc

:x: Forge suite twin_validator_test failure on testnet ==> a7f95bdafc94c901373e5b3241bb99ed76f140dc

twin validator : 4129 TPS, 7140 ms latency, 20200 ms p99 latency,no expired txns
Test Failed: TPS requirement failed. Average TPS 4129, minimum TPS requirement 6000
Trailing Log Lines:
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:238"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663193923-testnet","timestamp":"2022-09-14T22:33:14.834796Z","message":"Warmup stats: submitted: 5402 txn/s, committed: 5390 txn/s, expired: 12 txn/s, failed submission: 10 tnx/s, latency: 5874 ms, (p50: 5400 ms, p90: 9600 ms, p99: 18400 ms), latency samples: 80860"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:239"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663193923-testnet","timestamp":"2022-09-14T22:33:14.834807Z","message":"Test stats: submitted: 4129 txn/s, committed: 4129 txn/s, expired: 0 txn/s, failed submission: 0 tnx/s, latency: 7140 ms, (p50: 7100 ms, p90: 11200 ms, p99: 20200 ms), latency samples: 1114880"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:240"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663193923-testnet","timestamp":"2022-09-14T22:33:14.834817Z","message":"Cooldown stats: submitted: 4590 txn/s, committed: 4590 txn/s, expired: 0 txn/s, failed submission: 0 tnx/s, latency: 5795 ms, (p50: 5700 ms, p90: 8600 ms, p99: 14500 ms), latency samples: 68860"}
::error::TPS requirement failed. Average TPS 4129, minimum TPS requirement 6000
test twin validator ... FAILED
Error: TPS requirement failed. Average TPS 4129, minimum TPS requirement 6000
Test Statistics: 
twin validator : 4129 TPS, 7140 ms latency, 20200 ms p99 latency,no expired txns
Test Failed: TPS requirement failed. Average TPS 4129, minimum TPS requirement 6000


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:260"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663193923-testnet","timestamp":"2022-09-14T22:33:14.868056Z","message":"Deleting namespace forge-compat-pr-4121: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:368"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663193923-testnet","timestamp":"2022-09-14T22:33:14.868086Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4121"}

failures:
    twin validator

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge218-0       1/1     Running     0          11m
aptos-node-0-validator-0                1/1     Running     0          8m43s
aptos-node-1-fullnode-eforge218-0       1/1     Running     0          11m
aptos-node-1-validator-0                1/1     Running     0          6m52s
aptos-node-10-validator-0               1/1     Running     0          11m
aptos-node-11-validator-0               1/1     Running     0          11m
aptos-node-12-validator-0               1/1     Running     0          11m
aptos-node-13-validator-0               1/1     Running     0          11m
aptos-node-14-validator-0               1/1     Running     0          11m
aptos-node-15-validator-0               1/1     Running     0          11m
aptos-node-16-validator-0               1/1     Running     0          8m53s
aptos-node-17-validator-0               1/1     Running     0          7m2s
aptos-node-18-validator-0               1/1     Running     0          6m22s
aptos-node-19-validator-0               1/1     Running     0          5m42s
aptos-node-2-fullnode-eforge218-0       1/1     Running     0          11m
aptos-node-2-validator-0                1/1     Running     0          6m12s
aptos-node-3-fullnode-eforge218-0       1/1     Running     0          11m
aptos-node-3-validator-0                1/1     Running     0          5m31s
aptos-node-4-fullnode-eforge218-0       1/1     Running     0          11m
aptos-node-4-validator-0                1/1     Running     0          11m
aptos-node-5-fullnode-eforge218-0       1/1     Running     0          11m
aptos-node-5-validator-0                1/1     Running     0          11m
aptos-node-6-fullnode-eforge218-0       1/1     Running     0          11m
aptos-node-6-validator-0                1/1     Running     0          11m
aptos-node-7-fullnode-eforge218-0       1/1     Running     0          11m
aptos-node-7-validator-0                1/1     Running     0          11m
aptos-node-8-fullnode-eforge218-0       1/1     Running     0          11m
aptos-node-8-validator-0                1/1     Running     0          11m
aptos-node-9-fullnode-eforge218-0       1/1     Running     0          11m
aptos-node-9-validator-0                1/1     Running     0          11m
genesis-aptos-genesis-eforge218-fl2ss   0/1     Completed   0          13m

:x: Forge suite twin_validator_test failure on a7f95bdafc94c901373e5b3241bb99ed76f140dc

twin validator : 3727 TPS, 7811 ms latency, 18700 ms p99 latency,no expired txns
Test Failed: TPS requirement failed. Average TPS 3727, minimum TPS requirement 6000
Trailing Log Lines:
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:238"},"thread_name":"main","hostname":"forge-e2e-pr-4121-1663193915-a7f95bdafc94c901373e5b3241bb99ed76","timestamp":"2022-09-14T22:38:23.827096Z","message":"Warmup stats: submitted: 3878 txn/s, committed: 3873 txn/s, expired: 5 txn/s, failed submission: 4 tnx/s, latency: 8397 ms, (p50: 8700 ms, p90: 10800 ms, p99: 18900 ms), latency samples: 92960"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:239"},"thread_name":"main","hostname":"forge-e2e-pr-4121-1663193915-a7f95bdafc94c901373e5b3241bb99ed76","timestamp":"2022-09-14T22:38:23.827107Z","message":"Test stats: submitted: 3727 txn/s, committed: 3727 txn/s, expired: 0 txn/s, failed submission: 0 tnx/s, latency: 7811 ms, (p50: 7600 ms, p90: 11400 ms, p99: 18700 ms), latency samples: 1610400"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:240"},"thread_name":"main","hostname":"forge-e2e-pr-4121-1663193915-a7f95bdafc94c901373e5b3241bb99ed76","timestamp":"2022-09-14T22:38:23.827116Z","message":"Cooldown stats: submitted: 4122 txn/s, committed: 4122 txn/s, expired: 0 txn/s, failed submission: 0 tnx/s, latency: 7717 ms, (p50: 7600 ms, p90: 10800 ms, p99: 18400 ms), latency samples: 98940"}
::error::TPS requirement failed. Average TPS 3727, minimum TPS requirement 6000
test twin validator ... FAILED
Error: TPS requirement failed. Average TPS 3727, minimum TPS requirement 6000
Test Statistics: 
twin validator : 3727 TPS, 7811 ms latency, 18700 ms p99 latency,no expired txns
Test Failed: TPS requirement failed. Average TPS 3727, minimum TPS requirement 6000


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:260"},"thread_name":"main","hostname":"forge-e2e-pr-4121-1663193915-a7f95bdafc94c901373e5b3241bb99ed76","timestamp":"2022-09-14T22:38:23.864832Z","message":"Deleting namespace forge-e2e-pr-4121: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:368"},"thread_name":"main","hostname":"forge-e2e-pr-4121-1663193915-a7f95bdafc94c901373e5b3241bb99ed76","timestamp":"2022-09-14T22:38:23.864866Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4121"}

failures:
    twin validator

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge107-0       1/1     Running     0          16m
aptos-node-0-validator-0                1/1     Running     0          12m
aptos-node-1-fullnode-eforge107-0       1/1     Running     0          16m
aptos-node-1-validator-0                1/1     Running     0          11m
aptos-node-10-validator-0               1/1     Running     0          16m
aptos-node-11-validator-0               1/1     Running     0          16m
aptos-node-12-validator-0               1/1     Running     0          16m
aptos-node-13-validator-0               1/1     Running     0          16m
aptos-node-14-validator-0               1/1     Running     0          16m
aptos-node-15-validator-0               1/1     Running     0          16m
aptos-node-16-validator-0               1/1     Running     0          12m
aptos-node-17-validator-0               1/1     Running     0          11m
aptos-node-18-validator-0               1/1     Running     0          10m
aptos-node-19-validator-0               1/1     Running     0          10m
aptos-node-2-fullnode-eforge107-0       1/1     Running     0          16m
aptos-node-2-validator-0                1/1     Running     0          10m
aptos-node-3-fullnode-eforge107-0       1/1     Running     0          16m
aptos-node-3-validator-0                1/1     Running     0          9m52s
aptos-node-4-fullnode-eforge107-0       1/1     Running     0          16m
aptos-node-4-validator-0                1/1     Running     0          16m
aptos-node-5-fullnode-eforge107-0       1/1     Running     0          16m
aptos-node-5-validator-0                1/1     Running     0          16m
aptos-node-6-fullnode-eforge107-0       1/1     Running     0          16m
aptos-node-6-validator-0                1/1     Running     0          16m
aptos-node-7-fullnode-eforge107-0       1/1     Running     0          16m
aptos-node-7-validator-0                1/1     Running     0          16m
aptos-node-8-fullnode-eforge107-0       1/1     Running     0          16m
aptos-node-8-validator-0                1/1     Running     0          16m
aptos-node-9-fullnode-eforge107-0       1/1     Running     0          16m
aptos-node-9-validator-0                1/1     Running     0          16m
genesis-aptos-genesis-eforge107-dw8fv   0/1     Completed   0          18m

Forge is running suite twin_validator_test on 93776f80e860fecd5e25686088e16af5194a66b7

Forge is running suite twin_validator_test on testnet ==> 93776f80e860fecd5e25686088e16af5194a66b7

:x: Forge suite twin_validator_test failure on testnet ==> 93776f80e860fecd5e25686088e16af5194a66b7

Test Failed: Failed to mint accounts: Account 265cd28c2b0fe1f85acd0493c9a280874fb854799bc646cd1eaa6996cdfd41cd couldn't mint
Trailing Log Lines:
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/account_minter.rs:224"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663275462-testnet","timestamp":"2022-09-15T21:08:39.096172Z","message":"Creating and minting seeds accounts"}
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/account_minter.rs:131"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663275462-testnet","timestamp":"2022-09-15T21:08:40.141497Z","message":"Completed creating 30 seed accounts, each with 100101498999000 coins, had to retry 0 transactions"}
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/account_minter.rs:137"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663275462-testnet","timestamp":"2022-09-15T21:08:40.141519Z","message":"Minting additional 1500 accounts with 66733332666 coins each"}
{"level":"WARN","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/mod.rs:589"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663275462-testnet","timestamp":"2022-09-15T21:08:43.868460Z","message":"Failed to query sequence number for account LocalAccount { address: 313df553297789d6168ce03c92f93d8b62a68b576799e3c880bc9eb4ae626132, key: AccountKey { private_key: <elided secret for Ed25519PrivateKey>, public_key: Ed25519PublicKey(3562bc4d03dc956504bd47e96b6e5c1062d071a05f2c5803305078bdb8ed436d), authentication_key: AuthenticationKey([49, 61, 245, 83, 41, 119, 137, 214, 22, 140, 224, 60, 146, 249, 61, 139, 98, 166, 139, 87, 103, 153, 227, 200, 128, 188, 158, 180, 174, 98, 97, 50]) }, sequence_number: 0 } for instance Client { inner: Client { cookie_store: true, accepts: Accepts, proxies: [Proxy(System({}), None)], referer: true, default_headers: {\"accept\": \"*/*\", \"user-agent\": \"aptos-client-sdk-rust / 0.0.0\"}, timeout: 30s }, base_url: Url { scheme: \"http\", cannot_be_a_base: false, username: \"\", password: None, host: Some(Domain(\"aptos-node-0-fullnode.forge-compat-pr-4121.svc\")), port: Some(8080), path: \"/v1\", query: None, fragment: None }, version_path_base: \"/v1/\" } : Get accounts failed: API error Error(ResourceNotFound): Resource not found by Address(0x313df553297789d6168ce03c92f93d8b62a68b576799e3c880bc9eb4ae626132), Struct tag(0x1::account::Account) and Ledger version(3800)"}
::error::Failed to mint accounts: Account 265cd28c2b0fe1f85acd0493c9a280874fb854799bc646cd1eaa6996cdfd41cd couldn't mint
test twin validator ... FAILED
Error: Failed to mint accounts: Account 265cd28c2b0fe1f85acd0493c9a280874fb854799bc646cd1eaa6996cdfd41cd couldn't mint
Test Statistics: 
Test Failed: Failed to mint accounts: Account 265cd28c2b0fe1f85acd0493c9a280874fb854799bc646cd1eaa6996cdfd41cd couldn't mint


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:260"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663275462-testnet","timestamp":"2022-09-15T21:15:48.226436Z","message":"Deleting namespace forge-compat-pr-4121: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:368"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663275462-testnet","timestamp":"2022-09-15T21:15:48.226468Z","message":"aptos-node resources for Forge removed in namespace: forge-compat-pr-4121"}

failures:
    twin validator

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:

:x: Forge suite twin_validator_test failure on 93776f80e860fecd5e25686088e16af5194a66b7

twin validator : 3396 TPS, 8908 ms latency, 31400 ms p99 latency,no expired txns
Test Failed: TPS requirement failed. Average TPS 3396, minimum TPS requirement 6000
Trailing Log Lines:
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:238"},"thread_name":"main","hostname":"forge-e2e-pr-4121-1663275457-93776f80e860fecd5e25686088e16af519","timestamp":"2022-09-15T21:17:10.429243Z","message":"Warmup stats: submitted: 4300 txn/s, committed: 4299 txn/s, expired: 1 txn/s, failed submission: 1 tnx/s, latency: 8230 ms, (p50: 6600 ms, p90: 12900 ms, p99: 16900 ms), latency samples: 103180"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:239"},"thread_name":"main","hostname":"forge-e2e-pr-4121-1663275457-93776f80e860fecd5e25686088e16af519","timestamp":"2022-09-15T21:17:10.429258Z","message":"Test stats: submitted: 3396 txn/s, committed: 3396 txn/s, expired: 0 txn/s, failed submission: 0 tnx/s, latency: 8908 ms, (p50: 8100 ms, p90: 14200 ms, p99: 31400 ms), latency samples: 1467080"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:240"},"thread_name":"main","hostname":"forge-e2e-pr-4121-1663275457-93776f80e860fecd5e25686088e16af519","timestamp":"2022-09-15T21:17:10.429270Z","message":"Cooldown stats: submitted: 1250 txn/s, committed: 1250 txn/s, expired: 0 txn/s, failed submission: 0 tnx/s, latency: 11710 ms, (p50: 12000 ms, p90: 14500 ms, p99: 15100 ms), latency samples: 30020"}
::error::TPS requirement failed. Average TPS 3396, minimum TPS requirement 6000
test twin validator ... FAILED
Error: TPS requirement failed. Average TPS 3396, minimum TPS requirement 6000
Test Statistics: 
twin validator : 3396 TPS, 8908 ms latency, 31400 ms p99 latency,no expired txns
Test Failed: TPS requirement failed. Average TPS 3396, minimum TPS requirement 6000


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:260"},"thread_name":"main","hostname":"forge-e2e-pr-4121-1663275457-93776f80e860fecd5e25686088e16af519","timestamp":"2022-09-15T21:17:10.456074Z","message":"Deleting namespace forge-e2e-pr-4121: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:368"},"thread_name":"main","hostname":"forge-e2e-pr-4121-1663275457-93776f80e860fecd5e25686088e16af519","timestamp":"2022-09-15T21:17:10.456106Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4121"}

failures:
    twin validator

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                               READY   STATUS    RESTARTS   AGE
aptos-node-6-fullnode-eforge40-0   1/1     Running   0          16m
aptos-node-6-validator-0           1/1     Running   0          16m
aptos-node-7-fullnode-eforge40-0   1/1     Running   0          16m
aptos-node-7-validator-0           1/1     Running   0          16m
aptos-node-8-fullnode-eforge40-0   1/1     Running   0          16m
aptos-node-8-validator-0           1/1     Running   0          16m
aptos-node-9-fullnode-eforge40-0   1/1     Running   0          16m
aptos-node-9-validator-0           1/1     Running   0          16m

Forge is running suite twin_validator_test on testnet ==> 0de0228bcbe6d9076f5883de3a7c39c23d82095f

Forge is running suite twin_validator_test on 0de0228bcbe6d9076f5883de3a7c39c23d82095f

:white_check_mark: Forge suite twin_validator_test success on testnet ==> 0de0228bcbe6d9076f5883de3a7c39c23d82095f

twin validator : 7551 TPS, 3950 ms latency, 10800 ms p99 latency,no expired txns
Test Ok

:white_check_mark: Forge suite twin_validator_test success on 0de0228bcbe6d9076f5883de3a7c39c23d82095f

twin validator : 7192 TPS, 4130 ms latency, 10500 ms p99 latency,no expired txns
Test Ok

Forge is running suite twin_validator_test on 333bac74a58da4934c58e0d2c27ea32d2cec7412

:white_check_mark: Forge suite twin_validator_test success on testnet ==> 333bac74a58da4934c58e0d2c27ea32d2cec7412

twin validator : 7077 TPS, 4168 ms latency, 14400 ms p99 latency,no expired txns
Test Ok

:white_check_mark: Forge suite twin_validator_test success on 333bac74a58da4934c58e0d2c27ea32d2cec7412

twin validator : 6478 TPS, 4575 ms latency, 19600 ms p99 latency,no expired txns
Test Ok

github-actions[bot] avatar Sep 16 '22 02:09 github-actions[bot]

Forge is running suite twin_validator_test on testnet ==> 86ae0e7d3f2eb4245f51b0db11f1dd8b351f5f65

github-actions[bot] avatar Sep 16 '22 18:09 github-actions[bot]

Forge is running suite twin_validator_test on 86ae0e7d3f2eb4245f51b0db11f1dd8b351f5f65

github-actions[bot] avatar Sep 16 '22 18:09 github-actions[bot]

Forge is running suite land_blocking on 2244009c67b07978061e939e50b5efccbede9439

github-actions[bot] avatar Sep 16 '22 18:09 github-actions[bot]

Forge is running suite compat on testnet ==> 2244009c67b07978061e939e50b5efccbede9439

github-actions[bot] avatar Sep 16 '22 18:09 github-actions[bot]

:white_check_mark: Forge suite compat success on testnet ==> 2244009c67b07978061e939e50b5efccbede9439

Compatibility test results for testnet ==> 2244009c67b07978061e939e50b5efccbede9439 (PR)
1. Check liveness of validators at old version: testnet
compatibility::simple-validator-upgrade::liveness-check : 6444 TPS, 2688 ms latency, 3800 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 2244009c67b07978061e939e50b5efccbede9439
compatibility::simple-validator-upgrade::single-validator-upgrade : 6136 TPS, 3384 ms latency, 4300 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 2244009c67b07978061e939e50b5efccbede9439
compatibility::simple-validator-upgrade::half-validator-upgrade : 6068 TPS, 3156 ms latency, 4500 ms p99 latency,no expired txns
4. upgrading second batch to new version: 2244009c67b07978061e939e50b5efccbede9439
compatibility::simple-validator-upgrade::rest-validator-upgrade : 6264 TPS, 2859 ms latency, 4500 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet ==> 2244009c67b07978061e939e50b5efccbede9439 passed
Test Ok

github-actions[bot] avatar Sep 16 '22 18:09 github-actions[bot]

Forge is running suite land_blocking on 043913a7e6f3e0fcc2cbd16c804bb39101379b14

github-actions[bot] avatar Sep 16 '22 19:09 github-actions[bot]

Forge is running suite compat on testnet ==> 043913a7e6f3e0fcc2cbd16c804bb39101379b14

github-actions[bot] avatar Sep 16 '22 19:09 github-actions[bot]

:white_check_mark: Forge suite compat success on testnet ==> 043913a7e6f3e0fcc2cbd16c804bb39101379b14

Compatibility test results for testnet ==> 043913a7e6f3e0fcc2cbd16c804bb39101379b14 (PR)
1. Check liveness of validators at old version: testnet
compatibility::simple-validator-upgrade::liveness-check : 7890 TPS, 4798 ms latency, 6900 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 043913a7e6f3e0fcc2cbd16c804bb39101379b14
compatibility::simple-validator-upgrade::single-validator-upgrade : 6096 TPS, 6080 ms latency, 8100 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 043913a7e6f3e0fcc2cbd16c804bb39101379b14
compatibility::simple-validator-upgrade::half-validator-upgrade : 5140 TPS, 7253 ms latency, 11100 ms p99 latency,no expired txns
4. upgrading second batch to new version: 043913a7e6f3e0fcc2cbd16c804bb39101379b14
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7922 TPS, 4602 ms latency, 9000 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet ==> 043913a7e6f3e0fcc2cbd16c804bb39101379b14 passed
Test Ok

github-actions[bot] avatar Sep 16 '22 20:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on 043913a7e6f3e0fcc2cbd16c804bb39101379b14

performance benchmark with full nodes : 8065 TPS, 4914 ms latency, 8100 ms p99 latency,no expired txns
Test Ok

github-actions[bot] avatar Sep 16 '22 20:09 github-actions[bot]

Forge is running suite land_blocking on 4003c6aa4be95947dd981f9d40e033408411b709

github-actions[bot] avatar Sep 16 '22 20:09 github-actions[bot]

Forge is running suite compat on testnet ==> 4003c6aa4be95947dd981f9d40e033408411b709

github-actions[bot] avatar Sep 16 '22 20:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on 4003c6aa4be95947dd981f9d40e033408411b709

performance benchmark with full nodes : 8025 TPS, 4946 ms latency, 6600 ms p99 latency,no expired txns
Test Ok

github-actions[bot] avatar Sep 16 '22 21:09 github-actions[bot]

:x: Forge suite compat failure on testnet ==> 4003c6aa4be95947dd981f9d40e033408411b709

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:516"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663361557-testnet","timestamp":"2022-09-16T22:15:20.574378Z","message":"Node validator-3 unhealthy: Unknown error error sending request for url (http://aptos-node-3-validator.forge-compat-pr-4121.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:516"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663361557-testnet","timestamp":"2022-09-16T22:15:30.586530Z","message":"Node validator-3 unhealthy: Unknown error error sending request for url (http://aptos-node-3-validator.forge-compat-pr-4121.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/swarm.rs:516"},"thread_name":"main","hostname":"forge-compat-pr-4121-1663361557-testnet","timestamp":"2022-09-16T22:15:40.596003Z","message":"Node validator-3 unhealthy: Unknown error error sending request for url (http://aptos-node-3-validator.forge-compat-pr-4121.svc:8080/v1/): error trying to connect: dns error: failed to lookup address information: Name or service not known"}
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: query_sequence_numbers on Client { inner: Client { cookie_store: true, accepts: Accepts, proxies: [Proxy(System({}), None)], referer: true, default_headers: {"accept": "*/*", "user-agent": "aptos-client-sdk-rust / 0.0.0"}, timeout: 10s }, base_url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("aptos-node-0-validator.forge-compat-pr-4121.svc")), port: Some(8080), path: "/v1", query: None, fragment: None }, version_path_base: "/v1/" } for dd account failed: Get accounts failed: Unknown(error sending request for url (http://aptos-node-0-validator.forge-compat-pr-4121.svc:8080/v1/accounts/000000000000000000000000000000000000000000000000000000000a550c18): error trying to connect: dns error: failed to lookup address information: Name or service not known

Caused by:
    0: error trying to connect: dns error: failed to lookup address information: Name or service not known
    1: dns error: failed to lookup address information: Name or service not known
    2: failed to lookup address information: Name or service not known)', testsuite/forge/src/backend/k8s/mod.rs:167:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

github-actions[bot] avatar Sep 16 '22 22:09 github-actions[bot]

Forge is running suite land_blocking on bf1fbacd7153769a63b59ae47baa62c74449bcff

github-actions[bot] avatar Sep 16 '22 23:09 github-actions[bot]

Forge is running suite land_blocking on 7638715221eedd49d4fed4dd44bc497a9107c0f0

github-actions[bot] avatar Sep 17 '22 00:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on 7638715221eedd49d4fed4dd44bc497a9107c0f0

performance benchmark with full nodes : 7737 TPS, 5120 ms latency, 8300 ms p99 latency,(!) expired 180 out of 3342820 txns
Test Ok

github-actions[bot] avatar Sep 17 '22 00:09 github-actions[bot]

Forge is running suite land_blocking on ef4a92e280105b3054f3778c5999ed64cdc0d79b

github-actions[bot] avatar Sep 19 '22 10:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on ef4a92e280105b3054f3778c5999ed64cdc0d79b

performance benchmark with full nodes : 7776 TPS, 5106 ms latency, 7800 ms p99 latency,(!) expired 100 out of 3359620 txns
Test Ok

github-actions[bot] avatar Sep 19 '22 10:09 github-actions[bot]

Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> a52497dc881f8af625360ae114e6d1cf19907c0d

github-actions[bot] avatar Sep 19 '22 22:09 github-actions[bot]

Forge is running suite land_blocking on a52497dc881f8af625360ae114e6d1cf19907c0d

github-actions[bot] avatar Sep 19 '22 22:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on a52497dc881f8af625360ae114e6d1cf19907c0d

performance benchmark with full nodes : 7857 TPS, 5055 ms latency, 7200 ms p99 latency,(!) expired 20 out of 3394400 txns
Test Ok

github-actions[bot] avatar Sep 19 '22 22:09 github-actions[bot]

:white_check_mark: Forge suite compat success on 843b204dce971d98449b82624f4f684c7a18b991 ==> a52497dc881f8af625360ae114e6d1cf19907c0d

Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> a52497dc881f8af625360ae114e6d1cf19907c0d (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7594 TPS, 4830 ms latency, 7200 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: a52497dc881f8af625360ae114e6d1cf19907c0d
compatibility::simple-validator-upgrade::single-validator-upgrade : 5806 TPS, 6560 ms latency, 8900 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: a52497dc881f8af625360ae114e6d1cf19907c0d
compatibility::simple-validator-upgrade::half-validator-upgrade : 5546 TPS, 6497 ms latency, 8500 ms p99 latency,no expired txns
4. upgrading second batch to new version: a52497dc881f8af625360ae114e6d1cf19907c0d
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7461 TPS, 4817 ms latency, 10200 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for 843b204dce971d98449b82624f4f684c7a18b991 ==> a52497dc881f8af625360ae114e6d1cf19907c0d passed
Test Ok

github-actions[bot] avatar Sep 19 '22 22:09 github-actions[bot]

Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 1f2c104decf9004b40480e07fe7548ca8ff2a8eb

github-actions[bot] avatar Sep 19 '22 23:09 github-actions[bot]

Forge is running suite land_blocking on 1f2c104decf9004b40480e07fe7548ca8ff2a8eb

github-actions[bot] avatar Sep 19 '22 23:09 github-actions[bot]

Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 64afd0319eb531d94b477df196f59e2ebbd6094e

github-actions[bot] avatar Sep 20 '22 01:09 github-actions[bot]

Forge is running suite land_blocking on 64afd0319eb531d94b477df196f59e2ebbd6094e

github-actions[bot] avatar Sep 20 '22 01:09 github-actions[bot]

:white_check_mark: Forge suite compat success on 843b204dce971d98449b82624f4f684c7a18b991 ==> 64afd0319eb531d94b477df196f59e2ebbd6094e

Compatibility test results for 843b204dce971d98449b82624f4f684c7a18b991 ==> 64afd0319eb531d94b477df196f59e2ebbd6094e (PR)
1. Check liveness of validators at old version: 843b204dce971d98449b82624f4f684c7a18b991
compatibility::simple-validator-upgrade::liveness-check : 7825 TPS, 4856 ms latency, 6800 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 64afd0319eb531d94b477df196f59e2ebbd6094e
compatibility::simple-validator-upgrade::single-validator-upgrade : 5689 TPS, 6281 ms latency, 8900 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 64afd0319eb531d94b477df196f59e2ebbd6094e
compatibility::simple-validator-upgrade::half-validator-upgrade : 5182 TPS, 7453 ms latency, 9200 ms p99 latency,no expired txns
4. upgrading second batch to new version: 64afd0319eb531d94b477df196f59e2ebbd6094e
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7375 TPS, 4954 ms latency, 9600 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for 843b204dce971d98449b82624f4f684c7a18b991 ==> 64afd0319eb531d94b477df196f59e2ebbd6094e passed
Test Ok

github-actions[bot] avatar Sep 20 '22 01:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on 64afd0319eb531d94b477df196f59e2ebbd6094e

performance benchmark with full nodes : 7456 TPS, 5314 ms latency, 10200 ms p99 latency,(!) expired 240 out of 3221340 txns
Test Ok

github-actions[bot] avatar Sep 20 '22 01:09 github-actions[bot]

Forge is running suite land_blocking on 1c77d2119105480b8fd096943c80c348334f2739

github-actions[bot] avatar Sep 20 '22 13:09 github-actions[bot]