[test on forge][not for review] chain health block size backpressure

[igor-aptos]

Description

Test Plan

---

This change is 

[github-actions[bot]]

Forge is running suite three_region_simulation_with_different_node_speed on failpoints_5bb21bf11b48475482528b28902b7afe39236c67

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite three_region_simulation_with_different_node_speed success on failpoints_5bb21bf11b48475482528b28902b7afe39236c67

network::three-region-simulation : 3142 TPS, 1386 ms latency, 163300 ms p99 latency,(!) expired 2975234 out of 8010300 txns
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite three_region_simulation_with_different_node_speed on failpoints_fd38ac7a6b5e80b2a2527e8ed2d5eb578363dc8a

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite three_region_simulation_with_different_node_speed failure on failpoints_fd38ac7a6b5e80b2a2527e8ed2d5eb578363dc8a

Test Failed: start emitter job

Caused by:
    Failed to mint accounts: Account 3b8bdd6b6bbf242da4a3a48db65583b3d243c559041807dd247ce189e85de5c6 couldn't mint
    
    Caused by:
        Failed to wait for transaction: Unknown(Transaction expired. It is guaranteed it will not be committed on chain.), gas price: 100
Trailing Log Lines:
        Failed to wait for transaction: Unknown(Transaction expired. It is guaranteed it will not be committed on chain.), gas price: 100
Test Statistics: 
Test Failed: start emitter job

Caused by:
    Failed to mint accounts: Account 3b8bdd6b6bbf242da4a3a48db65583b3d243c559041807dd247ce189e85de5c6 couldn't mint
    
    Caused by:
        Failed to wait for transaction: Unknown(Transaction expired. It is guaranteed it will not be committed on chain.), gas price: 100


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-5543-1668212605-failpoints-fd38ac7a6b5e80b2a2527e8","timestamp":"2022-11-12T00:55:22.314358Z","message":"Deleting namespace forge-e2e-pr-5543: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-5543-1668212605-failpoints-fd38ac7a6b5e80b2a2527e8","timestamp":"2022-11-12T00:55:22.314392Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-5543"}

failures:
    network::three-region-simulation

test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed

Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge110-0       1/1     Running     0          27m
aptos-node-0-validator-0                1/1     Running     0          27m
aptos-node-1-fullnode-eforge110-0       1/1     Running     0          27m
aptos-node-1-validator-0                1/1     Running     0          27m
aptos-node-10-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-10-validator-0               1/1     Running     0          27m
aptos-node-11-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-11-validator-0               1/1     Running     0          27m
aptos-node-12-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-12-validator-0               1/1     Running     0          27m
aptos-node-13-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-13-validator-0               1/1     Running     0          27m
aptos-node-14-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-14-validator-0               1/1     Running     0          27m
aptos-node-15-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-15-validator-0               1/1     Running     0          27m
aptos-node-16-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-16-validator-0               1/1     Running     0          27m
aptos-node-17-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-17-validator-0               1/1     Running     0          27m
aptos-node-18-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-18-validator-0               1/1     Running     0          27m
aptos-node-19-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-19-validator-0               1/1     Running     0          27m
aptos-node-2-fullnode-eforge110-0       1/1     Running     0          27m
aptos-node-2-validator-0                1/1     Running     0          27m
aptos-node-20-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-20-validator-0               1/1     Running     0          27m
aptos-node-21-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-21-validator-0               1/1     Running     0          27m
aptos-node-22-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-22-validator-0               1/1     Running     0          27m
aptos-node-23-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-23-validator-0               1/1     Running     0          27m
aptos-node-24-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-24-validator-0               1/1     Running     0          27m
aptos-node-25-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-25-validator-0               1/1     Running     0          27m
aptos-node-26-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-26-validator-0               1/1     Running     0          27m
aptos-node-27-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-27-validator-0               1/1     Running     0          27m
aptos-node-28-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-28-validator-0               1/1     Running     0          27m
aptos-node-29-fullnode-eforge110-0      1/1     Running     0          27m
aptos-node-29-validator-0               1/1     Running     0          27m
aptos-node-3-fullnode-eforge110-0       1/1     Running     0          27m
aptos-node-3-validator-0                1/1     Running     0          27m
aptos-node-4-fullnode-eforge110-0       1/1     Running     0          27m
aptos-node-4-validator-0                1/1     Running     0          27m
aptos-node-5-fullnode-eforge110-0       1/1     Running     0          27m
aptos-node-5-validator-0                1/1     Running     0          27m
aptos-node-6-fullnode-eforge110-0       1/1     Running     0          27m
aptos-node-6-validator-0                1/1     Running     0          27m
aptos-node-7-fullnode-eforge110-0       1/1     Running     0          27m
aptos-node-7-validator-0                1/1     Running     0          27m
aptos-node-8-fullnode-eforge110-0       1/1     Running     0          27m
aptos-node-8-validator-0                1/1     Running     0          27m
aptos-node-9-fullnode-eforge110-0       1/1     Running     0          27m
aptos-node-9-validator-0                1/1     Running     0          27m
genesis-aptos-genesis-eforge110-26njg   0/1     Completed   0          31m

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite three_region_simulation_with_different_node_speed on failpoints_fd38ac7a6b5e80b2a2527e8ed2d5eb578363dc8a

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite three_region_simulation_with_different_node_speed success on failpoints_fd38ac7a6b5e80b2a2527e8ed2d5eb578363dc8a

network::three-region-simulation : 3780 TPS, 1580 ms latency, 238400 ms p99 latency,(!) expired 1939549 out of 7996100 txns
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite three_region_simulation_with_different_node_speed on failpoints_fd38ac7a6b5e80b2a2527e8ed2d5eb578363dc8a

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite three_region_simulation_with_different_node_speed success on failpoints_fd38ac7a6b5e80b2a2527e8ed2d5eb578363dc8a

network::three-region-simulation : 3663 TPS, 1547 ms latency, 213100 ms p99 latency,(!) expired 2096629 out of 7966060 txns
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite three_region_simulation_with_different_node_speed on failpoints_6b749ea2eb9d11f4a951b2f5a548409b0710558c

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite three_region_simulation_with_different_node_speed failure on failpoints_6b749ea2eb9d11f4a951b2f5a548409b0710558c

network::three-region-simulation : 135 TPS, 2175 ms latency, 94100 ms p99 latency,(!) expired 15586940 out of 16020140 txns
Test Failed: check for success

Caused by:
    TPS requirement failed. Average TPS 135, minimum TPS requirement 3000
Trailing Log Lines:

Caused by:
    TPS requirement failed. Average TPS 135, minimum TPS requirement 3000
Test Statistics: 
network::three-region-simulation : 135 TPS, 2175 ms latency, 94100 ms p99 latency,(!) expired 15586940 out of 16020140 txns
Test Failed: check for success

Caused by:
    TPS requirement failed. Average TPS 135, minimum TPS requirement 3000


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-5543-1668453964-failpoints-6b749ea2eb9d11f4a951b2f","timestamp":"2022-11-14T20:56:05.365495Z","message":"Deleting namespace forge-e2e-pr-5543: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-5543-1668453964-failpoints-6b749ea2eb9d11f4a951b2f","timestamp":"2022-11-14T20:56:05.365528Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-5543"}

failures:
    network::three-region-simulation

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                                   READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge93-0       1/1     Running     0          84m
aptos-node-0-validator-0               1/1     Running     0          84m
aptos-node-1-fullnode-eforge93-0       1/1     Running     0          84m
aptos-node-1-validator-0               1/1     Running     0          84m
aptos-node-10-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-10-validator-0              1/1     Running     0          84m
aptos-node-11-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-11-validator-0              1/1     Running     0          84m
aptos-node-12-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-12-validator-0              1/1     Running     0          84m
aptos-node-13-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-13-validator-0              1/1     Running     0          84m
aptos-node-14-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-14-validator-0              1/1     Running     0          84m
aptos-node-15-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-15-validator-0              1/1     Running     0          84m
aptos-node-16-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-16-validator-0              1/1     Running     0          84m
aptos-node-17-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-17-validator-0              1/1     Running     0          84m
aptos-node-18-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-18-validator-0              1/1     Running     0          84m
aptos-node-19-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-19-validator-0              1/1     Running     0          84m
aptos-node-2-fullnode-eforge93-0       1/1     Running     0          84m
aptos-node-2-validator-0               1/1     Running     0          84m
aptos-node-20-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-20-validator-0              1/1     Running     0          84m
aptos-node-21-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-21-validator-0              1/1     Running     0          84m
aptos-node-22-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-22-validator-0              1/1     Running     0          84m
aptos-node-23-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-23-validator-0              1/1     Running     0          84m
aptos-node-24-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-24-validator-0              1/1     Running     0          84m
aptos-node-25-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-25-validator-0              1/1     Running     0          84m
aptos-node-26-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-26-validator-0              1/1     Running     0          84m
aptos-node-27-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-27-validator-0              1/1     Running     0          84m
aptos-node-28-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-28-validator-0              1/1     Running     0          84m
aptos-node-29-fullnode-eforge93-0      1/1     Running     0          84m
aptos-node-29-validator-0              1/1     Running     0          84m
aptos-node-3-fullnode-eforge93-0       1/1     Running     0          84m
aptos-node-3-validator-0               1/1     Running     0          84m
aptos-node-4-fullnode-eforge93-0       1/1     Running     0          84m
aptos-node-4-validator-0               1/1     Running     0          84m
aptos-node-5-fullnode-eforge93-0       1/1     Running     0          84m
aptos-node-5-validator-0               1/1     Running     0          84m
aptos-node-6-fullnode-eforge93-0       1/1     Running     0          84m
aptos-node-6-validator-0               1/1     Running     0          84m
aptos-node-7-fullnode-eforge93-0       1/1     Running     0          84m
aptos-node-7-validator-0               1/1     Running     0          84m
aptos-node-8-fullnode-eforge93-0       1/1     Running     0          84m
aptos-node-8-validator-0               1/1     Running     0          84m
aptos-node-9-fullnode-eforge93-0       1/1     Running     0          84m
aptos-node-9-validator-0               1/1     Running     0          84m
genesis-aptos-genesis-eforge93-nnh67   0/1     Completed   0          89m

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite three_region_simulation_with_different_node_speed on failpoints_fe063358da0491976f02a80f1d4022c7ad38b3f5

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite three_region_simulation_with_different_node_speed failure on failpoints_fe063358da0491976f02a80f1d4022c7ad38b3f5

network::three-region-simulation : 2021 TPS, 1355 ms latency, 155000 ms p99 latency,(!) expired 9372984 out of 15849680 txns
Test Failed: check for success

Caused by:
    TPS requirement failed. Average TPS 2021, minimum TPS requirement 3000
Trailing Log Lines:

Caused by:
    TPS requirement failed. Average TPS 2021, minimum TPS requirement 3000
Test Statistics: 
network::three-region-simulation : 2021 TPS, 1355 ms latency, 155000 ms p99 latency,(!) expired 9372984 out of 15849680 txns
Test Failed: check for success

Caused by:
    TPS requirement failed. Average TPS 2021, minimum TPS requirement 3000


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-5543-1668476308-failpoints-fe063358da0491976f02a80","timestamp":"2022-11-15T02:54:45.056866Z","message":"Deleting namespace forge-e2e-pr-5543: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-5543-1668476308-failpoints-fe063358da0491976f02a80","timestamp":"2022-11-15T02:54:45.056892Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-5543"}

failures:
    network::three-region-simulation

test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed

Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge201-0       1/1     Running     0          72m
aptos-node-0-validator-0                1/1     Running     0          72m
aptos-node-1-fullnode-eforge201-0       1/1     Running     0          72m
aptos-node-1-validator-0                1/1     Running     0          72m
aptos-node-10-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-10-validator-0               1/1     Running     0          72m
aptos-node-11-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-11-validator-0               1/1     Running     0          72m
aptos-node-12-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-12-validator-0               1/1     Running     0          72m
aptos-node-13-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-13-validator-0               1/1     Running     0          72m
aptos-node-14-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-14-validator-0               1/1     Running     0          72m
aptos-node-15-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-15-validator-0               1/1     Running     0          72m
aptos-node-16-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-16-validator-0               1/1     Running     0          72m
aptos-node-17-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-17-validator-0               1/1     Running     0          72m
aptos-node-18-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-18-validator-0               1/1     Running     0          72m
aptos-node-19-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-19-validator-0               1/1     Running     0          72m
aptos-node-2-fullnode-eforge201-0       1/1     Running     0          72m
aptos-node-2-validator-0                1/1     Running     0          72m
aptos-node-20-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-20-validator-0               1/1     Running     0          72m
aptos-node-21-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-21-validator-0               1/1     Running     0          72m
aptos-node-22-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-22-validator-0               1/1     Running     0          72m
aptos-node-23-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-23-validator-0               1/1     Running     0          72m
aptos-node-24-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-24-validator-0               1/1     Running     0          72m
aptos-node-25-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-25-validator-0               1/1     Running     0          72m
aptos-node-26-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-26-validator-0               1/1     Running     0          72m
aptos-node-27-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-27-validator-0               1/1     Running     0          72m
aptos-node-28-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-28-validator-0               1/1     Running     0          72m
aptos-node-29-fullnode-eforge201-0      1/1     Running     0          72m
aptos-node-29-validator-0               1/1     Running     0          72m
aptos-node-3-fullnode-eforge201-0       1/1     Running     0          72m
aptos-node-3-validator-0                1/1     Running     0          72m
aptos-node-4-fullnode-eforge201-0       1/1     Running     0          72m
aptos-node-4-validator-0                1/1     Running     0          72m
aptos-node-5-fullnode-eforge201-0       1/1     Running     0          72m
aptos-node-5-validator-0                1/1     Running     0          72m
aptos-node-6-fullnode-eforge201-0       1/1     Running     0          72m
aptos-node-6-validator-0                1/1     Running     0          72m
aptos-node-7-fullnode-eforge201-0       1/1     Running     0          72m
aptos-node-7-validator-0                1/1     Running     0          72m
aptos-node-8-fullnode-eforge201-0       1/1     Running     0          72m
aptos-node-8-validator-0                1/1     Running     0          72m
aptos-node-9-fullnode-eforge201-0       1/1     Running     0          72m
aptos-node-9-validator-0                1/1     Running     0          72m
genesis-aptos-genesis-eforge201-9dv58   0/1     Completed   0          75m

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking