aptos-core icon indicating copy to clipboard operation
aptos-core copied to clipboard

Published 20 hours ago • verified publisher icon aptos-labs

[test on forge] load vs perf benchmark

Open igor-aptos opened this issue 3 years ago • 19 comments

Description

Test Plan

This change is Reviewable

igor-aptos avatar Sep 08 '22 20:09 igor-aptos

Forge is running suite land_blocking on caa7d76255d7293012bedd0ef4778283472b8a95

Forge is running suite compat on testnet ==> caa7d76255d7293012bedd0ef4778283472b8a95

:x: Forge suite compat failure on testnet ==> caa7d76255d7293012bedd0ef4778283472b8a95

Forge test runner terminated:
Trailing Log Lines:
unable to retrieve container logs for docker://4b7f2f847dec95dd754f3998935e4d9ca7e70191165504a73301fec7d182ecf9
Debugging output:

:white_check_mark: Forge suite land_blocking success on caa7d76255d7293012bedd0ef4778283472b8a95

Test Ok

Forge is running suite land_blocking on 24c18c51613e4348266d42790b1b3eeba05a93b0

Forge is running suite compat on testnet ==> 24c18c51613e4348266d42790b1b3eeba05a93b0

:x: Forge suite compat failure on testnet ==> 24c18c51613e4348266d42790b1b3eeba05a93b0

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'compat', testsuite/forge-cli/src/main.rs:184:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

:white_check_mark: Forge suite land_blocking success on 24c18c51613e4348266d42790b1b3eeba05a93b0

Test Ok

Forge is running suite land_blocking on f9d45f47fa82716efedaba9ccfc79d2f87a24d96

Forge is running suite compat on testnet ==> f9d45f47fa82716efedaba9ccfc79d2f87a24d96

:x: Forge suite compat failure on testnet ==> f9d45f47fa82716efedaba9ccfc79d2f87a24d96

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'compat', testsuite/forge-cli/src/main.rs:184:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

Forge is running suite compat on testnet ==> 96fba656a23677e2d264ec7eb1b2ebef8142ba49

Forge is running suite land_blocking on 96fba656a23677e2d264ec7eb1b2ebef8142ba49

:x: Forge suite compat failure on testnet ==> 96fba656a23677e2d264ec7eb1b2ebef8142ba49

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'compat', testsuite/forge-cli/src/main.rs:184:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

Forge is running suite land_blocking on 077f5aa3da40c26fd1bca805a9c77158baee7788

Forge is running suite compat on testnet ==> 077f5aa3da40c26fd1bca805a9c77158baee7788

:x: Forge suite compat failure on testnet ==> 077f5aa3da40c26fd1bca805a9c77158baee7788

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'compat', testsuite/forge-cli/src/main.rs:186:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

:white_check_mark: Forge suite land_blocking success on 077f5aa3da40c26fd1bca805a9c77158baee7788

Test Ok

Forge is running suite compat on testnet ==> e337007070b11669e02aa34eadd4a8f8f913dfa7

Forge is running suite land_blocking on e337007070b11669e02aa34eadd4a8f8f913dfa7

:x: Forge suite compat failure on testnet ==> e337007070b11669e02aa34eadd4a8f8f913dfa7

Forge test runner terminated:
Trailing Log Lines:
unable to retrieve container logs for docker://cf407044b561ad6c3deea3490839b98d97126084136ac1c81e09671021b3478c
Debugging output:

:white_check_mark: Forge suite land_blocking success on e337007070b11669e02aa34eadd4a8f8f913dfa7

Test Ok

Forge is running suite compat on testnet ==> 36b9b4eed7cda4212d54d3bbbbca818e6e36bfae

Forge is running suite land_blocking on 36b9b4eed7cda4212d54d3bbbbca818e6e36bfae

:x: Forge suite compat failure on testnet ==> 36b9b4eed7cda4212d54d3bbbbca818e6e36bfae

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'compat', testsuite/forge-cli/src/main.rs:187:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

:white_check_mark: Forge suite land_blocking success on 36b9b4eed7cda4212d54d3bbbbca818e6e36bfae

Test Ok

Forge is running suite compat on testnet ==> 0eed17d4eaa9089095870b5ba433d700790c5c60

Forge is running suite land_blocking on 0eed17d4eaa9089095870b5ba433d700790c5c60

:x: Forge suite compat failure on testnet ==> 0eed17d4eaa9089095870b5ba433d700790c5c60

Forge test runner terminated:
Trailing Log Lines:
unable to retrieve container logs for docker://1cb521198c9252b8366ddabe1f3428d922a64a44459a9e5769ddefafbf45c478
Debugging output:

:x: Forge suite land_blocking failure on 0eed17d4eaa9089095870b5ba433d700790c5c60

Test Failed: Failed to mint accounts: Account ffc75be8059611a17177eaad61ff03428a03edc3c1556c0bddc9832231bdf8b8 couldn't mint
Trailing Log Lines:
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/account_minter.rs:100"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663021137-0eed17d4eaa9089095870b5ba433d70079","timestamp":"2022-09-12T22:28:08.438423Z","message":"Root account current balance is 18446744073709551615, needed 38445770340942400 coins"}
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/account_minter.rs:224"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663021137-0eed17d4eaa9089095870b5ba433d70079","timestamp":"2022-09-12T22:28:08.438444Z","message":"Creating and minting seeds accounts"}
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/account_minter.rs:131"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663021137-0eed17d4eaa9089095870b5ba433d70079","timestamp":"2022-09-12T22:28:10.831425Z","message":"Completed creating 384 seed accounts, each with 100119193593600 coins, had to retry 0 transactions"}
{"level":"INFO","source":{"package":"transaction_emitter_lib","file":"crates/transaction-emitter-lib/src/emitter/account_minter.rs:137"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663021137-0eed17d4eaa9089095870b5ba433d70079","timestamp":"2022-09-12T22:28:10.831447Z","message":"Minting additional 19200 accounts with 5213541333 coins each"}
::error::Failed to mint accounts: Account ffc75be8059611a17177eaad61ff03428a03edc3c1556c0bddc9832231bdf8b8 couldn't mint
test continuous progress test ... FAILED
Error: Failed to mint accounts: Account ffc75be8059611a17177eaad61ff03428a03edc3c1556c0bddc9832231bdf8b8 couldn't mint
Test Statistics: 
Test Failed: Failed to mint accounts: Account ffc75be8059611a17177eaad61ff03428a03edc3c1556c0bddc9832231bdf8b8 couldn't mint


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:260"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663021137-0eed17d4eaa9089095870b5ba433d70079","timestamp":"2022-09-12T22:35:29.141079Z","message":"Deleting namespace forge-e2e-pr-3994: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:368"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663021137-0eed17d4eaa9089095870b5ba433d70079","timestamp":"2022-09-12T22:35:29.141109Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-3994"}

failures:
    continuous progress test

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:

Forge is running suite land_blocking on 0eed17d4eaa9089095870b5ba433d700790c5c60

:white_check_mark: Forge suite land_blocking success on 0eed17d4eaa9089095870b5ba433d700790c5c60

Test Ok

Forge is running suite land_blocking on 335c6757dbf1ca688f8685acce3e04710111cbb5

Forge is running suite compat on testnet ==> 335c6757dbf1ca688f8685acce3e04710111cbb5

:x: Forge suite compat failure on testnet ==> 335c6757dbf1ca688f8685acce3e04710111cbb5

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'compat', testsuite/forge-cli/src/main.rs:189:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

:white_check_mark: Forge suite land_blocking success on 335c6757dbf1ca688f8685acce3e04710111cbb5

performance benchmark with full nodes : 5556 TPS, 873 ms latency, 49200 ms p99 latency,(!) expired 5215800 out of 9716700 txns
Test Ok

Forge is running suite compat on testnet ==> ade8d4361e4a1ee39324fbf209dc36a468cfbe5d

Forge is running suite land_blocking on ade8d4361e4a1ee39324fbf209dc36a468cfbe5d

:x: Forge suite compat failure on testnet ==> ade8d4361e4a1ee39324fbf209dc36a468cfbe5d

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'compat', testsuite/forge-cli/src/main.rs:189:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

:x: Forge suite land_blocking failure on ade8d4361e4a1ee39324fbf209dc36a468cfbe5d

performance benchmark with full nodes : 4338 TPS, 2523 ms latency, 67500 ms p99 latency,(!) expired 6209211 out of 9723100 txns
Test Failed: TPS requirement failed. Average TPS 4338, minimum TPS requirement 5500
Trailing Log Lines:
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:279"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663178601-ade8d4361e4a1ee39324fbf209dc36a468","timestamp":"2022-09-14T18:27:35.186604Z","message":"Warmup stats: submitted: 11848 txn/s, committed: 11848 txn/s, expired: 0 txn/s, failed submission: 0 tnx/s, latency: 18194 ms, (p50: 18000 ms, p90: 32500 ms, p99: 37900 ms), latency samples: 10600"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:280"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663178601-ade8d4361e4a1ee39324fbf209dc36a468","timestamp":"2022-09-14T18:27:35.186614Z","message":"Test stats: submitted: 12003 txn/s, committed: 4338 txn/s, expired: 7665 txn/s, failed submission: 3422 tnx/s, latency: 63294 ms, (p50: 63900 ms, p90: 66300 ms, p99: 67500 ms), latency samples: 140086"}
{"level":"INFO","source":{"package":"testcases","file":"testsuite/testcases/src/lib.rs:281"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663178601-ade8d4361e4a1ee39324fbf209dc36a468","timestamp":"2022-09-14T18:27:35.186624Z","message":"Cooldown stats: submitted: 12024 txn/s, committed: 11180 txn/s, expired: 843 txn/s, failed submission: 5128 tnx/s, latency: 65010 ms, (p50: 64800 ms, p90: 65700 ms, p99: 67000 ms), latency samples: 8993"}
::error::TPS requirement failed. Average TPS 4338, minimum TPS requirement 5500
test performance benchmark with full nodes ... FAILED
Error: TPS requirement failed. Average TPS 4338, minimum TPS requirement 5500
Test Statistics: 
performance benchmark with full nodes : 4338 TPS, 2523 ms latency, 67500 ms p99 latency,(!) expired 6209211 out of 9723100 txns
Test Failed: TPS requirement failed. Average TPS 4338, minimum TPS requirement 5500


Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:260"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663178601-ade8d4361e4a1ee39324fbf209dc36a468","timestamp":"2022-09-14T18:27:35.269231Z","message":"Deleting namespace forge-e2e-pr-3994: Some(NamespaceStatus { phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:368"},"thread_name":"main","hostname":"forge-e2e-pr-3994-1663178601-ade8d4361e4a1ee39324fbf209dc36a468","timestamp":"2022-09-14T18:27:35.269258Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-3994"}

failures:
    performance benchmark with full nodes

test result: FAILED. 0 passed; 1 failed; 0 filtered out

Failed to run tests:
Tests Failed
Error: Tests Failed
Debugging output:
NAME                                    READY   STATUS      RESTARTS   AGE
aptos-node-0-fullnode-eforge171-0       1/1     Running     0          20m
aptos-node-0-validator-0                1/1     Running     0          20m
aptos-node-1-fullnode-eforge171-0       1/1     Running     0          20m
aptos-node-1-validator-0                1/1     Running     0          20m
aptos-node-10-validator-0               1/1     Running     0          20m
aptos-node-11-validator-0               1/1     Running     0          20m
aptos-node-12-validator-0               1/1     Running     0          20m
aptos-node-13-validator-0               1/1     Running     0          20m
aptos-node-14-validator-0               1/1     Running     0          20m
aptos-node-15-validator-0               1/1     Running     0          20m
aptos-node-16-validator-0               1/1     Running     0          20m
aptos-node-17-validator-0               1/1     Running     0          20m
aptos-node-18-validator-0               1/1     Running     0          20m
aptos-node-19-validator-0               1/1     Running     0          20m
aptos-node-2-fullnode-eforge171-0       1/1     Running     0          20m
aptos-node-2-validator-0                1/1     Running     0          20m
aptos-node-3-fullnode-eforge171-0       1/1     Running     0          20m
aptos-node-3-validator-0                1/1     Running     0          20m
aptos-node-4-fullnode-eforge171-0       1/1     Running     0          20m
aptos-node-4-validator-0                1/1     Running     0          20m
aptos-node-5-fullnode-eforge171-0       1/1     Running     0          20m
aptos-node-5-validator-0                1/1     Running     0          20m
aptos-node-6-fullnode-eforge171-0       1/1     Running     0          20m
aptos-node-6-validator-0                1/1     Running     0          20m
aptos-node-7-fullnode-eforge171-0       1/1     Running     0          20m
aptos-node-7-validator-0                1/1     Running     0          20m
aptos-node-8-fullnode-eforge171-0       1/1     Running     0          20m
aptos-node-8-validator-0                1/1     Running     0          20m
aptos-node-9-fullnode-eforge171-0       1/1     Running     0          20m
aptos-node-9-validator-0                1/1     Running     0          20m
genesis-aptos-genesis-eforge171-kw2rp   0/1     Completed   0          23m

github-actions[bot] avatar Sep 14 '22 18:09 github-actions[bot]

Forge is running suite land_blocking on 4203773838721bb5015903f90dd6dfdb94972482

github-actions[bot] avatar Sep 17 '22 00:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on 4203773838721bb5015903f90dd6dfdb94972482

Test Ok

github-actions[bot] avatar Sep 17 '22 04:09 github-actions[bot]

Forge is running suite land_blocking on 16b8ba8387b68d3a78841b6c6d5a8f1cc6ac641f

github-actions[bot] avatar Sep 19 '22 17:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on 16b8ba8387b68d3a78841b6c6d5a8f1cc6ac641f

Test Ok

github-actions[bot] avatar Sep 19 '22 18:09 github-actions[bot]

Forge is running suite land_blocking on 1d0852a54969dcbea0f7c47b64584ef553c5872e

github-actions[bot] avatar Sep 19 '22 20:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on 1d0852a54969dcbea0f7c47b64584ef553c5872e

Test Ok

github-actions[bot] avatar Sep 19 '22 22:09 github-actions[bot]

Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> e7709cd29adab3222bccf0306da9aa835f0e06ad

github-actions[bot] avatar Sep 30 '22 01:09 github-actions[bot]

Forge is running suite land_blocking on e7709cd29adab3222bccf0306da9aa835f0e06ad

github-actions[bot] avatar Sep 30 '22 01:09 github-actions[bot]

:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> e7709cd29adab3222bccf0306da9aa835f0e06ad

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'compat', testsuite/forge-cli/src/main.rs:190:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

github-actions[bot] avatar Sep 30 '22 01:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on e7709cd29adab3222bccf0306da9aa835f0e06ad

Test Ok

github-actions[bot] avatar Sep 30 '22 04:09 github-actions[bot]

Forge is running suite land_blocking on 0de4107d5d3321e6825ab56477efe760509ae3bd

github-actions[bot] avatar Sep 30 '22 20:09 github-actions[bot]

Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> 0de4107d5d3321e6825ab56477efe760509ae3bd

github-actions[bot] avatar Sep 30 '22 20:09 github-actions[bot]

:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> 0de4107d5d3321e6825ab56477efe760509ae3bd

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'compat', testsuite/forge-cli/src/main.rs:190:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

github-actions[bot] avatar Sep 30 '22 20:09 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on 0de4107d5d3321e6825ab56477efe760509ae3bd

Test Ok

github-actions[bot] avatar Sep 30 '22 23:09 github-actions[bot]

Forge is running suite land_blocking on ab16668559d8bdd8c458cce4b9b8dd5212605472

github-actions[bot] avatar Oct 06 '22 21:10 github-actions[bot]

Forge is running suite compat on 843b204dce971d98449b82624f4f684c7a18b991 ==> ab16668559d8bdd8c458cce4b9b8dd5212605472

github-actions[bot] avatar Oct 06 '22 21:10 github-actions[bot]

:x: Forge suite compat failure on 843b204dce971d98449b82624f4f684c7a18b991 ==> ab16668559d8bdd8c458cce4b9b8dd5212605472

Forge test runner terminated:
Trailing Log Lines:
thread 'main' panicked at 'compat', testsuite/forge-cli/src/main.rs:190:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Debugging output:

github-actions[bot] avatar Oct 06 '22 21:10 github-actions[bot]

:white_check_mark: Forge suite land_blocking success on ab16668559d8bdd8c458cce4b9b8dd5212605472

Test Ok

github-actions[bot] avatar Oct 07 '22 00:10 github-actions[bot]

closing out stale pr.

davidiw avatar Oct 28 '22 15:10 davidiw