aptos-core
aptos-core copied to clipboard
aptos-labs
[not to review] forge with block limit, gas schedule and aggressive backpressure
⏱️ 17h 54m total CI duration on this PR
| Job | Cumulative Duration | Recent Runs |
|---|---|---|
| forge-e2e-test / forge | 5h 38m | 🟥 🟥 🟥 🟥 🟥 (+1 more) |
| rust-unit-coverage | 4h 1m | 🟩 |
| windows-build | 2h 45m | 🟩 🟩 🟩 🟩 🟩 (+4 more) |
| rust-unit-tests | 2h 14m | 🟥 🟥 🟥 🟥 🟥 |
| rust-images / rust-all | 54m | 🟩 🟩 🟩 🟩 🟩 |
| rust-smoke-coverage | 36m | 🟩 |
| rust-lints | 24m | 🟥 🟥 🟥 🟥 🟥 |
| run-tests-main-branch | 23m | 🟩 🟩 🟩 🟩 🟩 |
| check | 21m | 🟩 🟩 ⬜ 🟩 🟩 (+2 more) |
| check-dynamic-deps | 17m | 🟩 🟩 🟩 🟩 🟩 (+4 more) |
| general-lints | 13m | 🟩 🟩 🟩 🟩 🟩 |
| semgrep/ci | 3m | 🟩 🟩 🟩 🟩 🟩 (+4 more) |
| file_change_determinator | 1m | 🟩 🟩 🟩 🟩 🟩 |
| file_change_determinator | 1m | 🟩 🟩 🟩 🟩 🟩 |
| permission-check | 56s | 🟩 🟩 🟩 🟩 🟩 (+4 more) |
| file_change_determinator | 54s | 🟩 🟩 ⬜ 🟩 🟩 (+2 more) |
| permission-check | 31s | 🟩 🟩 🟩 🟩 🟩 (+4 more) |
| permission-check | 15s | 🟩 🟩 ⬜ 🟩 🟩 (+2 more) |
| permission-check | 14s | 🟩 🟩 ⬜ 🟩 🟩 (+2 more) |
| upload-to-codecov | 14s | 🟩 |
| permission-check | 11s | 🟩 🟩 ⬜ 🟩 🟩 (+2 more) |
| determine-docker-build-metadata | 11s | 🟩 🟩 🟩 🟩 🟩 |
🚨 2 jobs on the last run were significantly faster/slower than expected
| Job | Duration | vs 7d avg | Delta |
|---|---|---|---|
| forge-e2e-test / forge | 1h 18m | 14m |  |
| rust-images / rust-all | 13m | 10m |  |
settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io
![trunk-io[bot] avatar](https://avatars.githubusercontent.com/in/120106?v=4 "Published by a pub.dev verified publisher"){kind=small}
Forge is running suite realistic_env_max_load on e9b7f10119a08c3bc00ea5707be1f694a01aa7ab
- Grafana dashboard (auto-refresh)
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
:x: Forge suite realistic_env_max_load failure on e9b7f10119a08c3bc00ea5707be1f694a01aa7ab
Forge test runner terminated:
Trailing Log Lines:
1: <unknown>
2: <unknown>
3: <unknown>
4: <unknown>
5: <unknown>
6: <unknown>
7: __libc_start_main
8: <unknown>', testsuite/forge/src/backend/k8s/mod.rs:191:10
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Debugging output:
Failed to get debugging output: error: You must be logged in to the server (Unauthorized)
- Grafana dashboard
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
Forge is running suite realistic_env_max_load on e9b7f10119a08c3bc00ea5707be1f694a01aa7ab
- Grafana dashboard (auto-refresh)
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite realistic_env_max_load failure on e9b7f10119a08c3bc00ea5707be1f694a01aa7ab
Forge test runner terminated:
Trailing Log Lines:
2: <unknown>
3: <unknown>
4: <unknown>
5: <unknown>
6: <unknown>
7: <unknown>
8: <unknown>
9: <unknown>
10: __libc_start_main
11: <unknown>
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-fullnode-eforge106-0 0/1 CrashLoopBackOff 2 (23s ago) 2m22s
aptos-node-0-validator-0 0/1 CrashLoopBackOff 3 (12s ago) 2m22s
aptos-node-1-fullnode-eforge106-0 0/1 CrashLoopBackOff 2 (24s ago) 2m22s
aptos-node-1-validator-0 0/1 CrashLoopBackOff 2 (25s ago) 2m22s
aptos-node-2-fullnode-eforge106-0 0/1 Error 3 (24s ago) 2m22s
aptos-node-2-validator-0 0/1 CrashLoopBackOff 2 (22s ago) 2m22s
aptos-node-3-validator-0 0/1 CrashLoopBackOff 3 (19s ago) 2m22s
aptos-node-4-validator-0 0/1 CrashLoopBackOff 2 (23s ago) 2m22s
aptos-node-5-validator-0 0/1 Error 3 (26s ago) 2m22s
aptos-node-6-validator-0 0/1 Error 3 (35s ago) 2m22s
genesis-aptos-genesis-eforge106-t6hqf 0/1 Completed 0 2m46s
- Grafana dashboard
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
:exclamation: No coverage uploaded for pull request base (
igor/test_block_gas_limit_and_calibration@fc8d3e4). Click here to learn what that means.
:exclamation: Current head e9b7f10 differs from pull request most recent head 6dd3fd8. Consider uploading reports for the commit 6dd3fd8 to get more accurate results
Additional details and impacted files
@@ Coverage Diff @@
## igor/test_block_gas_limit_and_calibration #11323 +/- ##
============================================================================
Coverage ? 69.1%
============================================================================
Files ? 2074
Lines ? 394679
Branches ? 0
============================================================================
Hits ? 273059
Misses ? 121620
Partials ? 0
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
![codecov[bot] avatar](https://avatars.githubusercontent.com/in/254?v=4 "Published by a pub.dev verified publisher"){kind=small}
[!WARNING] This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite. Learn more
Current dependencies on/for this PR:
- #11340
- #11323 👈
- #11312
- #11240 : 4 other dependent PRs (#11438 , #11538 , #11541 and 1 other)
- #11548
- #11547
- #11345 : 2 other dependent PRs (#11346 , #11537 )
main
This stack of pull requests is managed by Graphite.
Forge is running suite realistic_env_max_load on 8c5150342e590ae17ccb500f7d04acec8f9e876d
- Grafana dashboard (auto-refresh)
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite realistic_env_max_load failure on 8c5150342e590ae17ccb500f7d04acec8f9e876d
Forge test runner terminated:
Trailing Log Lines:
19: <unknown>
20: <unknown>
21: <unknown>
22: <unknown>
23: __libc_start_main
24: <unknown>', /aptos/crates/transaction-generator-lib/src/call_custom_modules.rs:163:14
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-11323-1702506776-8c5150342e590ae17ccb500f7d04acec8","timestamp":"2023-12-13T22:46:36.279167Z","message":"Deleting namespace forge-e2e-pr-11323: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-11323-1702506776-8c5150342e590ae17ccb500f7d04acec8","timestamp":"2023-12-13T22:46:36.279212Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-11323"}
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-fullnode-eforge45-0 1/1 Running 0 11m
aptos-node-0-validator-0 1/1 Running 0 11m
aptos-node-1-fullnode-eforge45-0 1/1 Running 0 11m
aptos-node-1-validator-0 1/1 Running 0 11m
aptos-node-2-fullnode-eforge45-0 1/1 Running 0 11m
aptos-node-2-validator-0 1/1 Running 0 11m
aptos-node-3-validator-0 1/1 Running 0 11m
aptos-node-4-validator-0 1/1 Running 0 11m
aptos-node-5-validator-0 1/1 Running 0 11m
aptos-node-6-validator-0 1/1 Running 0 11m
genesis-aptos-genesis-eforge45-zpbrt 0/1 Completed 0 12m
- Grafana dashboard
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
Forge is running suite realistic_env_max_load on bda116ad23732be892a1766dfa2df29712d60842
- Grafana dashboard (auto-refresh)
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite realistic_env_max_load failure on bda116ad23732be892a1766dfa2df29712d60842
Forge test runner terminated:
Trailing Log Lines:
19: <unknown>
20: <unknown>
21: <unknown>
22: <unknown>
23: __libc_start_main
24: <unknown>', /aptos/crates/transaction-generator-lib/src/call_custom_modules.rs:163:14
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-11323-1702512734-bda116ad23732be892a1766dfa2df2971","timestamp":"2023-12-14T00:24:53.901385Z","message":"Deleting namespace forge-e2e-pr-11323: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-11323-1702512734-bda116ad23732be892a1766dfa2df2971","timestamp":"2023-12-14T00:24:53.901426Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-11323"}
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-fullnode-eforge26-0 1/1 Running 0 12m
aptos-node-0-validator-0 1/1 Running 0 12m
aptos-node-1-fullnode-eforge26-0 1/1 Running 0 12m
aptos-node-1-validator-0 1/1 Running 0 12m
aptos-node-2-fullnode-eforge26-0 1/1 Running 0 12m
aptos-node-2-validator-0 1/1 Running 0 12m
aptos-node-3-validator-0 1/1 Running 0 12m
aptos-node-4-validator-0 1/1 Running 0 12m
aptos-node-5-validator-0 1/1 Running 0 12m
aptos-node-6-validator-0 1/1 Running 0 12m
genesis-aptos-genesis-eforge26-vxgbb 0/1 Completed 0 12m
- Grafana dashboard
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
Forge is running suite realistic_env_max_load on 511a31705919b804e5b42d3921c63153c0d89b6a
- Grafana dashboard (auto-refresh)
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite realistic_env_max_load failure on 511a31705919b804e5b42d3921c63153c0d89b6a
Forge test runner terminated:
Trailing Log Lines:
19: <unknown>
20: <unknown>
21: <unknown>
22: <unknown>
23: __libc_start_main
24: <unknown>', /aptos/crates/transaction-generator-lib/src/call_custom_modules.rs:163:14
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-11323-1702517017-511a31705919b804e5b42d3921c63153c","timestamp":"2023-12-14T01:36:53.128126Z","message":"Deleting namespace forge-e2e-pr-11323: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-11323-1702517017-511a31705919b804e5b42d3921c63153c","timestamp":"2023-12-14T01:36:53.128171Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-11323"}
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-fullnode-eforge181-0 1/1 Running 0 12m
aptos-node-0-validator-0 1/1 Running 0 12m
aptos-node-1-fullnode-eforge181-0 1/1 Running 0 12m
aptos-node-1-validator-0 1/1 Running 0 12m
aptos-node-2-fullnode-eforge181-0 1/1 Running 0 12m
aptos-node-2-validator-0 1/1 Running 0 12m
aptos-node-3-validator-0 1/1 Running 0 12m
aptos-node-4-validator-0 1/1 Running 0 12m
aptos-node-5-validator-0 1/1 Running 0 12m
aptos-node-6-validator-0 1/1 Running 0 12m
genesis-aptos-genesis-eforge181-zvgv2 0/1 Completed 0 13m
- Grafana dashboard
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
Forge is running suite realistic_env_max_load on 6dd3fd846daa8688fe9b8bee9b1319d8fd14e7de
- Grafana dashboard (auto-refresh)
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking
:x: Forge suite realistic_env_max_load failure on 6dd3fd846daa8688fe9b8bee9b1319d8fd14e7de
Workload | submitted/s | committed/s | expired/s | rejected/s | chain txn/s | latency | p50 lat | p90 lat | p99 lat | batch->pos | pos->prop | prop->order | order->commit | actual dur
0: CoinTransfer[50.0k] | 5851 | 5851 | 0 | 0 | 5851 | 8548 | 8500 | 8900 | 9300 | 0.200 | 0.134 | 0.487 | 0.387 | 177
1: NoOp(1000 modules)[50.0k] | 2548 | 2547 | 1 | 0 | 2553 | 19600 | 19200 | 22200 | 24700 | 0.197 | 1.133 | 1.104 | 2.774 | 177
2: ModifyGlobalResource[50.0k] | 3260 | 2874 | 386 | 0 | 2878 | 3664 | 3400 | 4800 | 8700 | 0.199 | 0.162 | 0.577 | 0.994 | 177
3: CreateObjects10WithPayload10k[50.0k] | 861 | 45 | 816 | 0 | 55 | 2824 | 2600 | 4400 | 5200 | 0.202 | 0.123 | 0.508 | 0.332 | 177
4: CreateObjectsConflict100WithPayload10k[50.0k] | 847 | 3 | 843 | 0 | 13 | 3490 | 3400 | 5100 | 5900 | 0.202 | 0.123 | 0.522 | 0.267 | 177
5: VectorPicture40[50.0k] | 3239 | 2851 | 388 | 0 | 2835 | 3432 | 3200 | 4900 | 8200 | 0.198 | 0.157 | 0.604 | 0.961 | 177
6: VectorPictureRead40[50.0k] | 5845 | 5845 | 0 | 0 | 5850 | 8554 | 8500 | 8900 | 9200 | 0.200 | 0.132 | 0.490 | 0.349 | 177
7: VectorPicture30k[50.0k] | 862 | 55 | 807 | 0 | 65 | 7240 | 6300 | 14700 | 17800 | 0.202 | 0.972 | 0.649 | 0.783 | 177
8: VectorPicture30k(20 modules)[50.0k] | 870 | 58 | 812 | 0 | 68 | 2775 | 2400 | 4600 | 5300 | 0.203 | 0.127 | 0.503 | 0.339 | 177
9: SmartTablePicture1MWith1KChange[50.0k] | 564 | 1 | 562 | 0 | 12 | 47205 | 48600 | 62600 | 83900 | 0.214 | 19.228 | 5.877 | 47.607 | 177
10: TokenV2AmbassadorMint[50.0k] | 1227 | 629 | 598 | 0 | 639 | 3463 | 3000 | 6700 | 9500 | 0.201 | 0.157 | 0.561 | 0.561 | 177
11: ph0: CreateBurnerAccounts[50.0k] | 5750 | 5748 | 1 | 0 | 5748 | 8839 | 8600 | 9200 | 15600 | 0.206 | 0.139 | 0.520 | 0.510 | 177
11: ph1: PublishPackage[50.0k] | 865 | 128 | 737 | 85 | 378 | 49842 | 53600 | 63900 | 71400 | 0.282 | 17.274 | 1.094 | 3.352 | 177
continuous traffic | submitted/s | committed/s | expired/s | rejected/s | latency | p50 lat | p90 lat | p99 lat
continuous idle | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0
continuous with traffic 0 | 10 | 10 | 0 | 0 | 2026 | 2100 | 2400 | 2700
continuous idle | 10 | 10 | 0 | 0 | 1342 | 1200 | 1500 | 1500
continuous with traffic 1 | 10 | 9 | 0 | 0 | 3384 | 2700 | 6000 | 6600
continuous idle | 10 | 10 | 0 | 0 | 1346 | 1200 | 1500 | 1500
continuous with traffic 2 | 10 | 10 | 0 | 0 | 2212 | 2100 | 3300 | 3900
continuous idle | 10 | 10 | 0 | 0 | 1363 | 1200 | 1500 | 1500
continuous with traffic 3 | 10 | 10 | 0 | 0 | 1405 | 1500 | 1500 | 1800
continuous idle | 9 | 9 | 0 | 0 | 1589 | 1500 | 1800 | 1800
continuous with traffic 4 | 10 | 10 | 0 | 0 | 1452 | 1500 | 1800 | 2100
continuous idle | 10 | 10 | 0 | 0 | 1363 | 1200 | 1800 | 1800
continuous with traffic 5 | 10 | 10 | 0 | 0 | 2151 | 1800 | 3300 | 3900
continuous idle | 10 | 10 | 0 | 0 | 1347 | 1200 | 1500 | 1500
continuous with traffic 6 | 10 | 10 | 0 | 0 | 1940 | 1800 | 2400 | 2400
continuous idle | 10 | 10 | 0 | 0 | 1259 | 1200 | 1200 | 1500
continuous with traffic 7 | 10 | 9 | 0 | 0 | 2742 | 1800 | 5100 | 6000
continuous idle | 10 | 10 | 0 | 0 | 1326 | 1200 | 1500 | 1500
continuous with traffic 8 | 10 | 10 | 0 | 0 | 1546 | 1500 | 1800 | 2400
continuous idle | 10 | 10 | 0 | 0 | 1439 | 1500 | 1500 | 1500
continuous with traffic 9 | 10 | 10 | 0 | 0 | 28210 | 29900 | 45800 | 79100
continuous idle | 9 | 9 | 0 | 0 | 1424 | 1500 | 1500 | 1500
continuous with traffic 10 | 10 | 10 | 0 | 0 | 1968 | 1500 | 3300 | 3900
continuous idle | 10 | 10 | 0 | 0 | 1439 | 1500 | 1500 | 1500
continuous with traffic 11 | 10 | 10 | 0 | 0 | 3568 | 2400 | 7500 | 9600
continuous idle | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Test Failed: TPS requirement for continuous idle failed. Average TPS 1, minimum TPS requirement 8. Full stats: committed: 1 txn/s, latency: 0 ms, (p50: 0 ms, p90: 0 ms, p99: 0 ms), latency samples: 0
Stack backtrace:
0: <unknown>
1: <unknown>
2: <unknown>
3: <unknown>
4: <unknown>
5: <unknown>
6: <unknown>
7: <unknown>
8: <unknown>
9: <unknown>
10: __libc_start_main
11: <unknown>
Trailing Log Lines:
3: <unknown>
4: <unknown>
5: <unknown>
6: <unknown>
7: <unknown>
8: <unknown>
9: <unknown>
10: __libc_start_main
11: <unknown>
Swarm logs can be found here: See fgi output for more information.
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:292"},"thread_name":"main","hostname":"forge-e2e-pr-11323-1702527388-6dd3fd846daa8688fe9b8bee9b1319d8f","timestamp":"2023-12-14T05:33:10.369911Z","message":"Deleting namespace forge-e2e-pr-11323: Some(NamespaceStatus { conditions: None, phase: Some(\"Terminating\") })"}
{"level":"INFO","source":{"package":"aptos_forge","file":"testsuite/forge/src/backend/k8s/cluster_helper.rs:400"},"thread_name":"main","hostname":"forge-e2e-pr-11323-1702527388-6dd3fd846daa8688fe9b8bee9b1319d8f","timestamp":"2023-12-14T05:33:10.369969Z","message":"aptos-node resources for Forge removed in namespace: forge-e2e-pr-11323"}
failures:
CompositeNetworkTest
test result: FAILED. 0 passed; 1 failed; 0 filtered out
Failed to run tests:
Tests Failed
Error: Tests Failed
Stack backtrace:
0: <unknown>
1: <unknown>
2: <unknown>
3: <unknown>
4: <unknown>
5: __libc_start_main
6: <unknown>
Debugging output:
NAME READY STATUS RESTARTS AGE
aptos-node-0-fullnode-eforge218-0 1/1 Running 0 74m
aptos-node-0-validator-0 1/1 Running 0 74m
aptos-node-1-fullnode-eforge218-0 1/1 Running 0 74m
aptos-node-1-validator-0 1/1 Running 0 74m
aptos-node-2-fullnode-eforge218-0 1/1 Running 0 74m
aptos-node-2-validator-0 1/1 Running 0 74m
aptos-node-3-validator-0 1/1 Running 0 74m
aptos-node-4-validator-0 1/1 Running 0 74m
aptos-node-5-validator-0 1/1 Running 0 74m
aptos-node-6-validator-0 1/1 Running 0 74m
genesis-aptos-genesis-eforge218-8tv6w 0/1 Completed 0 75m
- Grafana dashboard
- Humio Logs
- Axiom Logs
- Test runner output
- Test run is land-blocking