flow-go
flow-go copied to clipboard
onflow
FVM meter settings pre-fetch
In FVM RunV2(), meter setting values are read from state for each transaction, which is unnecessary. Now we move this out of tx execution up to per-collection level - meter setting values are not read once per transaction before tx.
NOTE: this is a breaking change.
- Ideally all EN and VN nodes should be deployed at the same time to avoid EN disagreement or VN verification failure.
- Flow emulator should also be updated afterwards to reflect this behavior change since it calls
FVM.RunV2()too.
Test
- [X] existing and newly added UT
- [X] running benchmark test on localnet and monitor everything
Performance
- Benchstat report in this PR
- pprof from localnet with the same set of benchmark testing
before optimization:
after optimization:
you will also need to prefetch and cache the state (GetEnvironmentMeterParameters doesn't do that currently). take a look at handler/program.go to see how it steals the state from the stateTransaction. you will need to do something similar here
you will also need to prefetch and cache the state (GetEnvironmentMeterParameters doesn't do that currently). take a look at handler/program.go to see how it steals the state from the stateTransaction. you will need to do something similar here
That should not be necessary since this is done per collection and a collection is also what gets sent to the versifiers. So as long as the touches happen once per collection, the VNs should get all the info needed. Unless i missed something...
Thanks for the suggestion. Now I have another question: how often, and when, are meter settings mutated? In this draft PR we are trying to read the state at the beginning of every collection. If this state can be mutated in the middle of one collection (like some tx can mutate), then we will not be able to do per collection state reading; but if this state is mutated even less frequent than collection level, we can do even less state polling. @janezpodhostnik
you will also need to prefetch and cache the state (GetEnvironmentMeterParameters doesn't do that currently). take a look at handler/program.go to see how it steals the state from the stateTransaction. you will need to do something similar here
To add up: to make sure meter setting state regs are sent over to VN for verification purposes, we will still need to touch those reg per collection.
Thanks for the suggestion. Now I have another question: how often, and when, are meter settings mutated? In this draft PR we are trying to read the state at the beginning of every collection. If this state can be mutated in the middle of one collection (like some tx can mutate), then we will not be able to do per collection state reading; but if this state is mutated even less frequent than collection level, we can do even less state polling. @janezpodhostnik
you will also need to prefetch and cache the state (GetEnvironmentMeterParameters doesn't do that currently). take a look at handler/program.go to see how it steals the state from the stateTransaction. you will need to do something similar here
you will also need to prefetch and cache the state (GetEnvironmentMeterParameters doesn't do that currently). take a look at handler/program.go to see how it steals the state from the stateTransaction. you will need to do something similar here
That should not be necessary since this is done per collection and a collection is also what gets sent to the versifiers. So as long as the touches happen once per collection, the VNs should get all the info needed. Unless i missed something...
you still need to capture the state once per collection, right now, GetEnvironmentMeterParameters discards that state?
Thanks for the suggestion. Now I have another question: how often, and when, are meter settings mutated? In this draft PR we are trying to read the state at the beginning of every collection. If this state can be mutated in the middle of one collection (like some tx can mutate), then we will not be able to do per collection state reading; but if this state is mutated even less frequent than collection level, we can do even less state polling. @janezpodhostnik
I've added the same question. sounds like it's safe to assume that the meter parameters are never modified within a block (it's updated at the end of a block). @janezpodhostnik can confirm
parameters are never modified within a block
That's both true and not :smile:. The transaction to modify the parameters can happen in the middle of a block (and in the middle of a collection), but if we move the read of the parameters to read once per collection, then the new parameters will be effective only in the next collections. This is fine, but is good to keep in mind.
FVM Benchstat comparison
This branch with compared with the base branch onflow:master commit a7a69d24bb28a4c909376d3b94f0d5e56dbffe25
The command (for i in {1..10}; do go test ./fvm ./engine/execution/computation --bench . --tags relic -shuffle=on --benchmem --run ^$; done) was used.
Collapsed results for better readability
| old.txt | new.txt | |||
|---|---|---|---|---|
| time/op | delta | |||
| pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64 | ||||
| RuntimeNFTBatchTransfer-2 | 133ms ± 9% | 130ms ± 3% | ~ | (p=0.167 n=9+8) |
| RuntimeTransaction/convert_int_to_string_and_concatenate_it-2 | 35.9ms ± 9% | 35.8ms ±12% | ~ | (p=0.853 n=10+10) |
| RuntimeTransaction/get_public_account-2 | 37.0ms ±14% | 35.1ms ±22% | ~ | (p=0.211 n=9+10) |
| RuntimeTransaction/get_account_and_get_balance-2 | 340ms ± 1% | 352ms ± 8% | ~ | (p=0.400 n=9+10) |
| RuntimeTransaction/get_account_and_get_available_balance-2 | 323ms ± 6% | 318ms ± 6% | ~ | (p=0.190 n=10+10) |
| RuntimeTransaction/get_account_and_get_storage_used-2 | 39.6ms ±12% | 38.6ms ±11% | ~ | (p=0.393 n=10+10) |
| RuntimeTransaction/get_account_and_get_storage_capacity-2 | 273ms ± 2% | 270ms ± 3% | ~ | (p=0.278 n=10+9) |
| RuntimeTransaction/get_signer_vault-2 | 42.4ms ± 5% | 40.8ms ± 9% | ~ | (p=0.247 n=10+10) |
| RuntimeTransaction/get_signer_receiver-2 | 53.0ms ± 3% | 52.8ms ± 9% | ~ | (p=0.631 n=10+10) |
| RuntimeTransaction/transfer_tokens-2 | 244ms ± 3% | 241ms ± 3% | ~ | (p=0.113 n=10+9) |
| RuntimeTransaction/load_and_save_empty_string_on_signers_address-2 | 42.3ms ±10% | 42.2ms ± 9% | ~ | (p=0.796 n=10+10) |
| RuntimeTransaction/load_and_save_long_string_on_signers_address-2 | 90.0ms ± 4% | 89.1ms ± 7% | ~ | (p=0.315 n=10+10) |
| RuntimeTransaction/create_new_account-2 | 1.00s ± 3% | 0.98s ± 1% | ~ | (p=0.093 n=9+8) |
| RuntimeTransaction/call_empty_contract_function-2 | 36.0ms ±12% | 34.5ms ±10% | ~ | (p=0.247 n=10+10) |
| RuntimeTransaction/emit_event-2 | 52.7ms ±10% | 51.6ms ± 6% | ~ | (p=0.393 n=10+10) |
| RuntimeTransaction/copy_array_from_storage-2 | 162ms ± 5% | 158ms ± 4% | ~ | (p=0.053 n=10+9) |
| RuntimeTransaction/borrow_array_from_storage-2 | 156ms ± 2% | 153ms ± 2% | −2.09% | (p=0.003 n=9+9) |
| pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64 | ||||
| ComputeBlock/16/cols/128/txes-2 | 5.82s ± 4% | 5.59s ± 4% | −3.95% | (p=0.003 n=10+9) |
| pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64 | ||||
| RuntimeTransaction/convert_int_to_string-2 | 35.6ms ±14% | 32.6ms ±15% | −8.47% | (p=0.015 n=10+10) |
| RuntimeTransaction/get_signer_address-2 | 33.2ms ±10% | 30.3ms ±12% | −8.64% | (p=0.009 n=10+10) |
| RuntimeTransaction/reference_tx-2 | 32.0ms ±12% | 28.8ms ± 9% | −9.99% | (p=0.003 n=10+9) |
| alloc/op | delta | |||
| pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64 | ||||
| RuntimeNFTBatchTransfer-2 | 57.6MB ± 3% | 57.6MB ± 4% | ~ | (p=0.739 n=10+10) |
| RuntimeTransaction/convert_int_to_string-2 | 38.2MB ± 6% | 36.5MB ±10% | ~ | (p=0.280 n=10+10) |
| RuntimeTransaction/convert_int_to_string_and_concatenate_it-2 | 38.0MB ± 9% | 38.4MB ± 8% | ~ | (p=0.579 n=10+10) |
| RuntimeTransaction/get_public_account-2 | 38.9MB ± 6% | 38.1MB ± 9% | ~ | (p=0.436 n=10+10) |
| RuntimeTransaction/get_account_and_get_balance-2 | 130MB ± 1% | 129MB ± 4% | ~ | (p=0.133 n=9+10) |
| RuntimeTransaction/get_account_and_get_available_balance-2 | 113MB ± 2% | 113MB ± 3% | ~ | (p=0.739 n=10+10) |
| RuntimeTransaction/get_account_and_get_storage_used-2 | 39.2MB ±10% | 38.9MB ± 8% | ~ | (p=0.631 n=10+10) |
| RuntimeTransaction/get_account_and_get_storage_capacity-2 | 106MB ± 4% | 106MB ± 4% | ~ | (p=0.631 n=10+10) |
| RuntimeTransaction/get_signer_vault-2 | 40.1MB ± 6% | 39.3MB ± 7% | ~ | (p=0.393 n=10+10) |
| RuntimeTransaction/get_signer_receiver-2 | 42.0MB ± 3% | 42.5MB ± 8% | ~ | (p=0.133 n=9+10) |
| RuntimeTransaction/transfer_tokens-2 | 88.8MB ± 3% | 88.5MB ± 3% | ~ | (p=0.739 n=10+10) |
| RuntimeTransaction/load_and_save_empty_string_on_signers_address-2 | 40.0MB ± 9% | 40.1MB ± 5% | ~ | (p=0.684 n=10+10) |
| RuntimeTransaction/load_and_save_long_string_on_signers_address-2 | 56.7MB ± 3% | 56.0MB ± 3% | ~ | (p=0.113 n=9+10) |
| RuntimeTransaction/create_new_account-2 | 201MB ± 1% | 201MB ± 3% | ~ | (p=0.063 n=10+10) |
| RuntimeTransaction/call_empty_contract_function-2 | 37.7MB ±11% | 37.5MB ± 7% | ~ | (p=0.853 n=10+10) |
| RuntimeTransaction/emit_event-2 | 42.2MB ± 9% | 41.7MB ± 4% | ~ | (p=0.579 n=10+10) |
| RuntimeTransaction/borrow_array_from_storage-2 | 69.2MB ± 4% | 69.1MB ± 3% | ~ | (p=0.579 n=10+10) |
| RuntimeTransaction/copy_array_from_storage-2 | 82.2MB ± 2% | 82.2MB ± 2% | ~ | (p=0.720 n=10+9) |
| RuntimeTransaction/reference_tx-2 | 37.5MB ± 7% | 35.6MB ± 7% | −5.01% | (p=0.023 n=10+10) |
| pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64 | ||||
| ComputeBlock/16/cols/128/txes-2 | 1.34GB ± 1% | 1.27GB ± 1% | −5.05% | (p=0.000 n=10+10) |
| pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64 | ||||
| RuntimeTransaction/get_signer_address-2 | 37.6MB ± 7% | 35.6MB ± 7% | −5.23% | (p=0.010 n=9+10) |
| allocs/op | delta | |||
| pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64 | ||||
| RuntimeTransaction/create_new_account-2 | 2.67M ± 0% | 2.67M ± 0% | −0.23% | (p=0.000 n=10+10) |
| RuntimeTransaction/get_account_and_get_balance-2 | 1.51M ± 0% | 1.51M ± 0% | −0.41% | (p=0.000 n=10+10) |
| RuntimeTransaction/get_account_and_get_available_balance-2 | 1.40M ± 0% | 1.39M ± 0% | −0.43% | (p=0.000 n=10+9) |
| RuntimeTransaction/get_account_and_get_storage_capacity-2 | 1.24M ± 0% | 1.23M ± 0% | −0.49% | (p=0.000 n=10+10) |
| RuntimeTransaction/transfer_tokens-2 | 934k ± 0% | 928k ± 0% | −0.64% | (p=0.000 n=9+10) |
| RuntimeTransaction/borrow_array_from_storage-2 | 368k ± 0% | 362k ± 0% | −1.65% | (p=0.000 n=10+10) |
| RuntimeTransaction/copy_array_from_storage-2 | 324k ± 0% | 318k ± 0% | −1.87% | (p=0.000 n=8+8) |
| RuntimeNFTBatchTransfer-2 | 290k ± 0% | 285k ± 0% | −2.01% | (p=0.000 n=8+9) |
| RuntimeTransaction/load_and_save_long_string_on_signers_address-2 | 231k ± 0% | 225k ± 0% | −2.62% | (p=0.000 n=10+10) |
| RuntimeTransaction/get_signer_receiver-2 | 203k ± 0% | 197k ± 0% | −2.98% | (p=0.000 n=10+10) |
| RuntimeTransaction/emit_event-2 | 136k ± 0% | 130k ± 0% | −4.45% | (p=0.000 n=10+10) |
| RuntimeTransaction/load_and_save_empty_string_on_signers_address-2 | 129k ± 0% | 123k ± 0% | −4.70% | (p=0.000 n=10+10) |
| RuntimeTransaction/get_account_and_get_storage_used-2 | 128k ± 0% | 122k ± 0% | −4.73% | (p=0.000 n=10+10) |
| RuntimeTransaction/get_signer_vault-2 | 127k ± 0% | 121k ± 0% | −4.79% | (p=0.000 n=10+10) |
| RuntimeTransaction/convert_int_to_string_and_concatenate_it-2 | 107k ± 0% | 101k ± 0% | −5.65% | (p=0.000 n=10+10) |
| RuntimeTransaction/get_public_account-2 | 107k ± 0% | 101k ± 0% | −5.66% | (p=0.000 n=10+10) |
| RuntimeTransaction/call_empty_contract_function-2 | 95.6k ± 0% | 89.5k ± 0% | −6.34% | (p=0.000 n=10+10) |
| RuntimeTransaction/convert_int_to_string-2 | 92.9k ± 0% | 86.8k ± 0% | −6.53% | (p=0.000 n=10+10) |
| pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64 | ||||
| ComputeBlock/16/cols/128/txes-2 | 20.5M ± 0% | 19.2M ± 0% | −6.61% | (p=0.000 n=10+10) |
| pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64 | ||||
| RuntimeTransaction/get_signer_address-2 | 83.9k ± 0% | 77.8k ± 0% | −7.24% | (p=0.000 n=10+10) |
| RuntimeTransaction/reference_tx-2 | 78.7k ± 0% | 72.6k ± 0% | −7.72% | (p=0.000 n=10+10) |
| computation | delta | |||
| pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64 | ||||
| RuntimeTransaction/reference_tx-2 | 202 ± 0% | 202 ± 0% | ~ | (all equal) |
| RuntimeTransaction/convert_int_to_string-2 | 402 ± 0% | 402 ± 0% | ~ | (all equal) |
| RuntimeTransaction/convert_int_to_string_and_concatenate_it-2 | 502 ± 0% | 502 ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_signer_address-2 | 302 ± 0% | 302 ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_public_account-2 | 402 ± 0% | 402 ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_account_and_get_balance-2 | 1.00k ± 0% | 1.00k ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_account_and_get_available_balance-2 | 2.60k ± 0% | 2.60k ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_account_and_get_storage_used-2 | 402 ± 0% | 402 ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_account_and_get_storage_capacity-2 | 1.30k ± 0% | 1.30k ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_signer_vault-2 | 402 ± 0% | 402 ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_signer_receiver-2 | 602 ± 0% | 602 ± 0% | ~ | (all equal) |
| RuntimeTransaction/transfer_tokens-2 | 3.50k ± 0% | 3.50k ± 0% | ~ | (all equal) |
| RuntimeTransaction/load_and_save_empty_string_on_signers_address-2 | 602 ± 0% | 602 ± 0% | ~ | (all equal) |
| RuntimeTransaction/load_and_save_long_string_on_signers_address-2 | 602 ± 0% | 602 ± 0% | ~ | (all equal) |
| RuntimeTransaction/create_new_account-2 | 202 ± 0% | 202 ± 0% | ~ | (all equal) |
| RuntimeTransaction/call_empty_contract_function-2 | 402 ± 0% | 402 ± 0% | ~ | (all equal) |
| RuntimeTransaction/emit_event-2 | 602 ± 0% | 602 ± 0% | ~ | (all equal) |
| RuntimeTransaction/borrow_array_from_storage-2 | 2.60k ± 0% | 2.60k ± 0% | ~ | (all equal) |
| RuntimeTransaction/copy_array_from_storage-2 | 2.60k ± 0% | 2.60k ± 0% | ~ | (all equal) |
| interactions | delta | |||
| pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64 | ||||
| RuntimeTransaction/reference_tx-2 | 44.3k ± 0% | 44.3k ± 0% | ~ | (all equal) |
| RuntimeTransaction/convert_int_to_string-2 | 44.3k ± 0% | 44.3k ± 0% | ~ | (all equal) |
| RuntimeTransaction/convert_int_to_string_and_concatenate_it-2 | 44.3k ± 0% | 44.3k ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_signer_address-2 | 44.3k ± 0% | 44.3k ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_public_account-2 | 44.3k ± 0% | 44.3k ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_account_and_get_balance-2 | 16.7M ± 0% | 16.7M ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_account_and_get_available_balance-2 | 5.13M ± 0% | 5.13M ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_account_and_get_storage_used-2 | 44.3k ± 0% | 44.3k ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_account_and_get_storage_capacity-2 | 5.13M ± 0% | 5.13M ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_signer_vault-2 | 44.5k ± 0% | 44.5k ± 0% | ~ | (all equal) |
| RuntimeTransaction/get_signer_receiver-2 | 44.9k ± 0% | 44.9k ± 0% | ~ | (all equal) |
| RuntimeTransaction/transfer_tokens-2 | 45.2k ± 0% | 45.2k ± 0% | ~ | (all equal) |
| RuntimeTransaction/load_and_save_empty_string_on_signers_address-2 | 44.4k ± 0% | 44.4k ± 0% | ~ | (all equal) |
| RuntimeTransaction/load_and_save_long_string_on_signers_address-2 | 47.2k ± 0% | 47.2k ± 0% | ~ | (all equal) |
| RuntimeTransaction/create_new_account-2 | 8.39M ± 0% | 8.39M ± 0% | ~ | (all equal) |
| RuntimeTransaction/call_empty_contract_function-2 | 44.5k ± 0% | 44.5k ± 0% | ~ | (all equal) |
| RuntimeTransaction/emit_event-2 | 44.5k ± 0% | 44.5k ± 0% | ~ | (all equal) |
| RuntimeTransaction/borrow_array_from_storage-2 | 49.7k ± 0% | 49.7k ± 0% | ~ | (all equal) |
| RuntimeTransaction/copy_array_from_storage-2 | 49.7k ± 0% | 49.7k ± 0% | ~ | (all equal) |
| us/tx | delta | |||
| pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64 | ||||
| ComputeBlock/16/cols/128/txes-2 | 2.84k ± 4% | 2.73k ± 4% | −3.95% | (p=0.002 n=10+9) |
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Codecov Report
Merging #3240 (f71f407) into master (a7a69d2) will decrease coverage by
1.18%. The diff coverage is77.46%.
@@ Coverage Diff @@
## master #3240 +/- ##
==========================================
- Coverage 55.16% 53.98% -1.19%
==========================================
Files 736 589 -147
Lines 67164 50727 -16437
==========================================
- Hits 37053 27384 -9669
+ Misses 27072 21113 -5959
+ Partials 3039 2230 -809
| Flag | Coverage Δ | |
|---|---|---|
| unittests | 53.98% <77.46%> (-1.19%) |
:arrow_down: |
Flags with carried forward coverage won't be shown. Click here to find out more.
| Impacted Files | Coverage Δ | |
|---|---|---|
| module/chunks/chunkVerifier.go | 69.46% <70.00%> (-0.12%) |
:arrow_down: |
| fvm/executionParameters.go | 80.74% <71.42%> (-2.59%) |
:arrow_down: |
| fvm/fvm.go | 69.16% <85.71%> (+3.42%) |
:arrow_up: |
| fvm/context.go | 60.62% <100.00%> (+1.27%) |
:arrow_up: |
| insecure/corruptible/network.go | 48.65% <0.00%> (-0.60%) |
:arrow_down: |
| engine/execution/computation/computer/computer.go | ||
| ...ngine/consensus/approvals/aggregated_signatures.go | ||
| engine/access/rpc/backend/backend_block_headers.go | ||
| engine/access/rest/request/transaction.go | ||
| ... and 148 more |
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.
after discussion we decided to use state-replay cache to do cached state reads while keeping the original reg touch records.