[test on forge] changing quorum for fixing voting history

[igor-aptos]

Description

Test Plan

---

This change is 

[github-actions[bot]]

Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 91c0b1ca09765b61cab6726d88740eab9c191902

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite land_blocking on 91c0b1ca09765b61cab6726d88740eab9c191902

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 91c0b1ca09765b61cab6726d88740eab9c191902

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'explicit panic', testsuite/forge-cli/src/main.rs:190:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite land_blocking failure on 91c0b1ca09765b61cab6726d88740eab9c191902

Forge test runner terminated:
Trailing Log Lines:
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:244"},"thread_name":"main","hostname":"forge-e2e-pr-4984-1665544693-91c0b1ca09765b61cab6726d88740eab9c","timestamp":"2022-10-12T03:25:35.552671Z","message":"84s warmup finished"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/consensus_reliability_tests.rs:77"},"thread_name":"main","hostname":"forge-e2e-pr-4984-1665544693-91c0b1ca09765b61cab6726d88740eab9c","timestamp":"2022-10-12T03:25:35.650956Z","message":"Always healthy 0 nodes, every cycle having 6 nodes out of 20 down, rotating 2 each cycle, expecting first 0 validators to have 10x larger stake"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/consensus_reliability_tests.rs:125"},"thread_name":"main","hostname":"forge-e2e-pr-4984-1665544693-91c0b1ca09765b61cab6726d88740eab9c","timestamp":"2022-10-12T03:25:35.788098Z","message":"For cycle 0 down nodes: {3, 4, 5, 0, 2, 1}"}
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: validator-3

Caused by:
    API error Error(WebFrameworkError): Failpoints are not enabled at a feature level', testsuite/forge/src/test_utils/consensus_utils.rs:307:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4984-1665544693-91c0b1ca09765b61cab6726d88740eab9c","timestamp":"2022-10-12T03:25:35.851898Z","message":"Deleting namespace forge-e2e-pr-4984: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4984-1665544693-91c0b1ca09765b61cab6726d88740eab9c","timestamp":"2022-10-12T03:25:35.851936Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4984"}
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge207-0       1/1     Running     0          3m53s
aptos-node-0-validator-0                1/1     Running     0          3m53s
aptos-node-1-fullnode-eforge207-0       1/1     Running     0          3m53s
aptos-node-1-validator-0                1/1     Running     0          3m53s
aptos-node-10-validator-0               1/1     Running     0          3m52s
aptos-node-11-validator-0               1/1     Running     0          3m53s
aptos-node-12-validator-0               1/1     Running     0          3m53s
aptos-node-13-validator-0               1/1     Running     0          3m53s
aptos-node-14-validator-0               1/1     Running     0          3m53s
aptos-node-15-validator-0               1/1     Running     0          3m53s
aptos-node-16-validator-0               1/1     Running     0          3m53s
aptos-node-17-validator-0               1/1     Running     0          3m53s
aptos-node-18-validator-0               1/1     Running     0          3m53s
aptos-node-19-validator-0               1/1     Running     0          3m53s
aptos-node-2-validator-0                1/1     Running     0          3m53s
aptos-node-3-validator-0                1/1     Running     0          3m53s
aptos-node-4-validator-0                1/1     Running     0          3m53s
aptos-node-5-validator-0                1/1     Running     0          3m53s
aptos-node-6-validator-0                1/1     Running     0          3m52s
aptos-node-7-validator-0                1/1     Running     0          3m53s
aptos-node-8-validator-0                1/1     Running     0          3m53s
aptos-node-9-validator-0                1/1     Running     0          3m53s
genesis-aptos-genesis-eforge207-4fnrq   0/1     Completed   0          6m41s

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite land_blocking on failpoints_cc18df4d2aff934c093580bdd121250e6d608240

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite land_blocking success on failpoints_cc18df4d2aff934c093580bdd121250e6d608240

changing working quorum test : 100 TPS, 546 ms latency, 80500 ms p99 latency,no expired txns
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite land_blocking on failpoints_491a5af1fb67ce12077f32194c9fd708547818d5

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite land_blocking success on failpoints_491a5af1fb67ce12077f32194c9fd708547818d5

changing working quorum test : 100 TPS, 560 ms latency, 80500 ms p99 latency,no expired txns
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite land_blocking on failpoints_86ff5663fbeec29990e996c0526df6c227954379

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:x: Forge suite land_blocking failure on failpoints_86ff5663fbeec29990e996c0526df6c227954379

changing working quorum test : 55 TPS, 2972 ms latency, 136900 ms p99 latency,(!) expired 46695 out of 105620 txns
Test Failed: check for success

Caused by:
    TPS requirement failed. Average TPS 55, minimum TPS requirement 70
Trailing Log Lines:

Caused by:
    TPS requirement failed. Average TPS 55, minimum TPS requirement 70
Test Statistics: 
changing working quorum test : 55 TPS, 2972 ms latency, 136900 ms p99 latency,(!) expired 46695 out of 105620 txns
Test Failed: check for success

Caused by:
    TPS requirement failed. Average TPS 55, minimum TPS requirement 70


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-4984-1665691875-failpoints-86ff5663fbeec29990e996c","timestamp":"2022-10-13T20:39:16.957695Z","message":"Deleting namespace forge-e2e-pr-4984: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:388"},"thread_name":"main","hostname":"forge-e2e-pr-4984-1665691875-failpoints-86ff5663fbeec29990e996c","timestamp":"2022-10-13T20:39:16.957719Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-4984"}

failures:
    changing working quorum test

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge133-0       1/1     Running     0          24m
aptos-node-0-validator-0                1/1     Running     0          24m
aptos-node-1-fullnode-eforge133-0       1/1     Running     0          24m
aptos-node-1-validator-0                1/1     Running     0          24m
aptos-node-10-validator-0               1/1     Running     0          24m
aptos-node-11-validator-0               1/1     Running     0          24m
aptos-node-12-validator-0               1/1     Running     0          24m
aptos-node-13-validator-0               1/1     Running     0          24m
aptos-node-14-validator-0               1/1     Running     0          24m
aptos-node-15-validator-0               1/1     Running     0          24m
aptos-node-16-validator-0               1/1     Running     0          24m
aptos-node-17-validator-0               1/1     Running     0          24m
aptos-node-18-validator-0               1/1     Running     0          24m
aptos-node-19-validator-0               1/1     Running     0          24m
aptos-node-2-validator-0                1/1     Running     0          24m
aptos-node-3-validator-0                1/1     Running     0          24m
aptos-node-4-validator-0                1/1     Running     0          24m
aptos-node-5-validator-0                1/1     Running     0          24m
aptos-node-6-validator-0                1/1     Running     0          24m
aptos-node-7-validator-0                1/1     Running     0          24m
aptos-node-8-validator-0                1/1     Running     0          24m
aptos-node-9-validator-0                1/1     Running     0          24m
genesis-aptos-genesis-eforge133-pwlfs   0/1     Completed   0          27m

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite land_blocking on failpoints_eb3c9a90a56f2548ff60417445dcc2ce4d40f2e0

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite land_blocking success on failpoints_eb3c9a90a56f2548ff60417445dcc2ce4d40f2e0

changing working quorum test : 100 TPS, 79 ms latency, 13500 ms p99 latency,no expired txns
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite land_blocking on failpoints_703d787afacfb45b1ba71edd0cd9b99aa1954853

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite land_blocking success on failpoints_703d787afacfb45b1ba71edd0cd9b99aa1954853

changing working quorum test : 100 TPS, 121 ms latency, 17100 ms p99 latency,no expired txns
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

Forge is running suite land_blocking on failpoints_9cfd5061a8cfa68d418a49a6bd35d11cd810bb8d

• Grafana dashboard (auto-refresh)
• Humio Logs
• Test runner output
• Test run is land-blocking

[github-actions[bot]]

:white_check_mark: Forge suite land_blocking success on failpoints_9cfd5061a8cfa68d418a49a6bd35d11cd810bb8d

changing working quorum test : 100 TPS, 134 ms latency, 16000 ms p99 latency,no expired txns
Test Ok

• Grafana dashboard
• Humio Logs
• Test runner output
• Test run is land-blocking

[davidiw]

looks stale