Forge is running with 049b385a10aa633d2cef35909f1982c91addcbaf
:x: Forge test perf regression on 049b385a10aa633d2cef35909f1982c91addcbaf
Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:84"},"thread_name":"main","hostname":"forge-pr-3779-1662137222-049b385a10aa633d2cef35909f1982c91addcb","timestamp":"2022-09-02T16:50:36.952634Z","message":"Genesis done"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:314"},"thread_name":"main","hostname":"forge-pr-3779-1662137222-049b385a10aa633d2cef35909f1982c91addcb","timestamp":"2022-09-02T16:50:36.952676Z","message":"[\"upgrade\", \"--install\", \"--create-namespace\", \"--namespace\", \"forge-pr-3779\", \"aptos-node\", \"terraform/helm/aptos-node\", \"--reuse-values\", \"--history-max\", \"2\", \"-f\", \"/tmp/.tmpqUtGJB/aptos-node_status.json\", \"-f\", \"/tmp/.tmpqUtGJB/aptos-node-values.yaml\", \"--set\", \"podSecurityPolicy=false\"]"}
Release "aptos-node" does not exist. Installing it now.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:260"},"thread_name":"main","hostname":"forge-pr-3779-1662137222-049b385a10aa633d2cef35909f1982c91addcb","timestamp":"2022-09-02T16:50:38.393648Z","message":"Deleting namespace forge-pr-3779: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:368"},"thread_name":"main","hostname":"forge-pr-3779-1662137222-049b385a10aa633d2cef35909f1982c91addcb","timestamp":"2022-09-02T16:50:38.393677Z","message":"aptos-node resources for Forge removed in namespace: forge-pr-3779"}
Failed to run tests:
Upgrade not completed: Error: persistentvolumeclaims "aptos-node-7-validator-eforge113" is forbidden: rpc error: code = ResourceExhausted desc = etcdserver: throttle: too many requests
Error: Upgrade not completed: Error: persistentvolumeclaims "aptos-node-7-validator-eforge113" is forbidden: rpc error: code = ResourceExhausted desc = etcdserver: throttle: too many requests
Debugging output:
NAME READY STATUS RESTARTS AGE
genesis-aptos-genesis-eforge113-v6g7f 0/1 Completed 0 2m34s
Forge is running with 049b385a10aa633d2cef35909f1982c91addcbaf
:white_check_mark: Forge test success on 049b385a10aa633d2cef35909f1982c91addcbaf
performance benchmark with full nodes : 6863 TPS, 4350 ms latency, 6000 ms p99 latency,no expired txns
Test Ok
Forge is running suite compat on testnet ==> 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
Forge is running suite land_blocking on 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
:white_check_mark: Forge suite compat success on testnet ==> 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
Compatibility test results for testnet ==> 99cb8d25516313bc87c65ab7456ebbb7d735b4fd (PR)
1. Check liveness of validators at old version: testnet
compatibility::simple-validator-upgrade::liveness-check : 6368 TPS, 2581 ms latency, 3700 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
compatibility::simple-validator-upgrade::single-validator-upgrade : 7392 TPS, 2872 ms latency, 4200 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
compatibility::simple-validator-upgrade::half-validator-upgrade : 6316 TPS, 3321 ms latency, 3900 ms p99 latency,no expired txns
4. upgrading second batch to new version: 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
compatibility::simple-validator-upgrade::rest-validator-upgrade : 8748 TPS, 2810 ms latency, 4600 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet ==> 99cb8d25516313bc87c65ab7456ebbb7d735b4fd passed
Test Ok
:white_check_mark: Forge suite land_blocking success on 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
performance benchmark with full nodes : 7614 TPS, 3909 ms latency, 5700 ms p99 latency,no expired txns
Test Ok
Forge is running suite land_blocking on 47bc68e477b7ae7ac64b61d20aa97682e3026a30
:white_check_mark: Forge suite compat success on testnet ==> 47bc68e477b7ae7ac64b61d20aa97682e3026a30
Compatibility test results for testnet ==> 47bc68e477b7ae7ac64b61d20aa97682e3026a30 (PR)
1. Check liveness of validators at old version: testnet
compatibility::simple-validator-upgrade::liveness-check : 6180 TPS, 2704 ms latency, 3900 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 47bc68e477b7ae7ac64b61d20aa97682e3026a30
compatibility::simple-validator-upgrade::single-validator-upgrade : 7256 TPS, 3087 ms latency, 4200 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 47bc68e477b7ae7ac64b61d20aa97682e3026a30
compatibility::simple-validator-upgrade::half-validator-upgrade : 6236 TPS, 3294 ms latency, 4400 ms p99 latency,no expired txns
4. upgrading second batch to new version: 47bc68e477b7ae7ac64b61d20aa97682e3026a30
compatibility::simple-validator-upgrade::rest-validator-upgrade : 5760 TPS, 2701 ms latency, 3700 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet ==> 47bc68e477b7ae7ac64b61d20aa97682e3026a30 passed
Test Ok
:white_check_mark: Forge suite land_blocking success on 47bc68e477b7ae7ac64b61d20aa97682e3026a30
performance benchmark with full nodes : 7663 TPS, 3867 ms latency, 5700 ms p99 latency,no expired txns
Test Ok
Forge is running suite land_blocking on 8183b1d32a4282fa6969744425106095ae744fd0
Forge is running suite compat on testnet ==> 8183b1d32a4282fa6969744425106095ae744fd0
Forge is running suite land_blocking on 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
Forge is running suite compat on testnet ==> 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
:white_check_mark: Forge suite land_blocking success on 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
performance benchmark with full nodes : 7762 TPS, 5116 ms latency, 7900 ms p99 latency,no expired txns
Test Ok
:white_check_mark: Forge suite compat success on testnet ==> 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
Compatibility test results for testnet ==> 0b0f7dd50edde1d7f0876feb5926c6ebedf66210 (PR)
1. Check liveness of validators at old version: testnet
compatibility::simple-validator-upgrade::liveness-check : 7940 TPS, 4559 ms latency, 6600 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
compatibility::simple-validator-upgrade::single-validator-upgrade : 5278 TPS, 7137 ms latency, 9400 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
compatibility::simple-validator-upgrade::half-validator-upgrade : 5360 TPS, 7034 ms latency, 9500 ms p99 latency,no expired txns
4. upgrading second batch to new version: 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7586 TPS, 4751 ms latency, 8400 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet ==> 0b0f7dd50edde1d7f0876feb5926c6ebedf66210 passed
Test Ok
Forge is running suite compat on testnet ==> a5c45042cd86a8f5f04adcf984b0a69516836939
Forge is running suite land_blocking on a5c45042cd86a8f5f04adcf984b0a69516836939
:white_check_mark: Forge suite compat success on testnet ==> a5c45042cd86a8f5f04adcf984b0a69516836939
Compatibility test results for testnet ==> a5c45042cd86a8f5f04adcf984b0a69516836939 (PR)
1. Check liveness of validators at old version: testnet
compatibility::simple-validator-upgrade::liveness-check : 8312 TPS, 4460 ms latency, 7800 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: a5c45042cd86a8f5f04adcf984b0a69516836939
compatibility::simple-validator-upgrade::single-validator-upgrade : 5758 TPS, 6341 ms latency, 8000 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: a5c45042cd86a8f5f04adcf984b0a69516836939
compatibility::simple-validator-upgrade::half-validator-upgrade : 5718 TPS, 6672 ms latency, 8200 ms p99 latency,no expired txns
4. upgrading second batch to new version: a5c45042cd86a8f5f04adcf984b0a69516836939
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7708 TPS, 4649 ms latency, 7600 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet ==> a5c45042cd86a8f5f04adcf984b0a69516836939 passed
Test Ok
Forge is running with 049b385a10aa633d2cef35909f1982c91addcbaf
:x: Forge test perf regression on 049b385a10aa633d2cef35909f1982c91addcbaf
Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:84"},"thread_name":"main","hostname":"forge-pr-3779-1662137222-049b385a10aa633d2cef35909f1982c91addcb","timestamp":"2022-09-02T16:50:36.952634Z","message":"Genesis done"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:314"},"thread_name":"main","hostname":"forge-pr-3779-1662137222-049b385a10aa633d2cef35909f1982c91addcb","timestamp":"2022-09-02T16:50:36.952676Z","message":"[\"upgrade\", \"--install\", \"--create-namespace\", \"--namespace\", \"forge-pr-3779\", \"aptos-node\", \"terraform/helm/aptos-node\", \"--reuse-values\", \"--history-max\", \"2\", \"-f\", \"/tmp/.tmpqUtGJB/aptos-node_status.json\", \"-f\", \"/tmp/.tmpqUtGJB/aptos-node-values.yaml\", \"--set\", \"podSecurityPolicy=false\"]"}
Release "aptos-node" does not exist. Installing it now.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:260"},"thread_name":"main","hostname":"forge-pr-3779-1662137222-049b385a10aa633d2cef35909f1982c91addcb","timestamp":"2022-09-02T16:50:38.393648Z","message":"Deleting namespace forge-pr-3779: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:368"},"thread_name":"main","hostname":"forge-pr-3779-1662137222-049b385a10aa633d2cef35909f1982c91addcb","timestamp":"2022-09-02T16:50:38.393677Z","message":"aptos-node resources for Forge removed in namespace: forge-pr-3779"}
Failed to run tests:
Upgrade not completed: Error: persistentvolumeclaims "aptos-node-7-validator-eforge113" is forbidden: rpc error: code = ResourceExhausted desc = etcdserver: throttle: too many requests
Error: Upgrade not completed: Error: persistentvolumeclaims "aptos-node-7-validator-eforge113" is forbidden: rpc error: code = ResourceExhausted desc = etcdserver: throttle: too many requests
Debugging output:
NAME READY STATUS RESTARTS AGE
genesis-aptos-genesis-eforge113-v6g7f 0/1 Completed 0 2m34s
Forge is running with 049b385a10aa633d2cef35909f1982c91addcbaf
:white_check_mark: Forge test success on 049b385a10aa633d2cef35909f1982c91addcbaf
performance benchmark with full nodes : 6863 TPS, 4350 ms latency, 6000 ms p99 latency,no expired txns
Test Ok
Forge is running suite compat on testnet ==> 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
Forge is running suite land_blocking on 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
:white_check_mark: Forge suite compat success on testnet ==> 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
Compatibility test results for testnet ==> 99cb8d25516313bc87c65ab7456ebbb7d735b4fd (PR)
1. Check liveness of validators at old version: testnet
compatibility::simple-validator-upgrade::liveness-check : 6368 TPS, 2581 ms latency, 3700 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
compatibility::simple-validator-upgrade::single-validator-upgrade : 7392 TPS, 2872 ms latency, 4200 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
compatibility::simple-validator-upgrade::half-validator-upgrade : 6316 TPS, 3321 ms latency, 3900 ms p99 latency,no expired txns
4. upgrading second batch to new version: 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
compatibility::simple-validator-upgrade::rest-validator-upgrade : 8748 TPS, 2810 ms latency, 4600 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet ==> 99cb8d25516313bc87c65ab7456ebbb7d735b4fd passed
Test Ok
:white_check_mark: Forge suite land_blocking success on 99cb8d25516313bc87c65ab7456ebbb7d735b4fd
performance benchmark with full nodes : 7614 TPS, 3909 ms latency, 5700 ms p99 latency,no expired txns
Test Ok
Forge is running suite land_blocking on 47bc68e477b7ae7ac64b61d20aa97682e3026a30
:white_check_mark: Forge suite compat success on testnet ==> 47bc68e477b7ae7ac64b61d20aa97682e3026a30
Compatibility test results for testnet ==> 47bc68e477b7ae7ac64b61d20aa97682e3026a30 (PR)
1. Check liveness of validators at old version: testnet
compatibility::simple-validator-upgrade::liveness-check : 6180 TPS, 2704 ms latency, 3900 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 47bc68e477b7ae7ac64b61d20aa97682e3026a30
compatibility::simple-validator-upgrade::single-validator-upgrade : 7256 TPS, 3087 ms latency, 4200 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 47bc68e477b7ae7ac64b61d20aa97682e3026a30
compatibility::simple-validator-upgrade::half-validator-upgrade : 6236 TPS, 3294 ms latency, 4400 ms p99 latency,no expired txns
4. upgrading second batch to new version: 47bc68e477b7ae7ac64b61d20aa97682e3026a30
compatibility::simple-validator-upgrade::rest-validator-upgrade : 5760 TPS, 2701 ms latency, 3700 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet ==> 47bc68e477b7ae7ac64b61d20aa97682e3026a30 passed
Test Ok
:white_check_mark: Forge suite land_blocking success on 47bc68e477b7ae7ac64b61d20aa97682e3026a30
performance benchmark with full nodes : 7663 TPS, 3867 ms latency, 5700 ms p99 latency,no expired txns
Test Ok
Forge is running suite land_blocking on 8183b1d32a4282fa6969744425106095ae744fd0
Forge is running suite compat on testnet ==> 8183b1d32a4282fa6969744425106095ae744fd0
Forge is running suite land_blocking on 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
Forge is running suite compat on testnet ==> 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
:white_check_mark: Forge suite land_blocking success on 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
performance benchmark with full nodes : 7762 TPS, 5116 ms latency, 7900 ms p99 latency,no expired txns
Test Ok
:white_check_mark: Forge suite compat success on testnet ==> 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
Compatibility test results for testnet ==> 0b0f7dd50edde1d7f0876feb5926c6ebedf66210 (PR)
1. Check liveness of validators at old version: testnet
compatibility::simple-validator-upgrade::liveness-check : 7940 TPS, 4559 ms latency, 6600 ms p99 latency,no expired txns
2. Upgrading first Validator to new version: 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
compatibility::simple-validator-upgrade::single-validator-upgrade : 5278 TPS, 7137 ms latency, 9400 ms p99 latency,no expired txns
3. Upgrading rest of first batch to new version: 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
compatibility::simple-validator-upgrade::half-validator-upgrade : 5360 TPS, 7034 ms latency, 9500 ms p99 latency,no expired txns
4. upgrading second batch to new version: 0b0f7dd50edde1d7f0876feb5926c6ebedf66210
compatibility::simple-validator-upgrade::rest-validator-upgrade : 7586 TPS, 4751 ms latency, 8400 ms p99 latency,no expired txns
5. check swarm health
Compatibility test for testnet ==> 0b0f7dd50edde1d7f0876feb5926c6ebedf66210 passed
Test Ok
Forge is running suite compat on testnet ==> a5c45042cd86a8f5f04adcf984b0a69516836939
Forge is running suite land_blocking on a5c45042cd86a8f5f04adcf984b0a69516836939
:white_check_mark: Forge suite land_blocking success on a5c45042cd86a8f5f04adcf984b0a69516836939
performance benchmark with full nodes : 8159 TPS, 4858 ms latency, 7500 ms p99 latency,no expired txns
Test Ok