Execution data verification

[m4ksio]

[github-actions[bot]]

FVM Benchstat comparison

This branch with compared with the base branch onflow:master commit d96331d4bea34a185c43e73aa269d0fe0225810a

The command (for i in {1..7}; do go test ./fvm ./engine/execution/computation --bench . --tags relic -shuffle=on --benchmem --run ^$; done) was used.

Collapsed results for better readability

	old.txt	new.txt
	time/op		delta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-2	34.5ms ± 4%	34.2ms ± 5%	~	(p=0.628 n=6+7)
RuntimeTransaction/convert_int_to_string-2	42.2ms ±16%	36.6ms ± 3%	~	(p=0.101 n=7+6)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2	39.0ms ±11%	40.7ms ±20%	~	(p=0.535 n=7+7)
RuntimeTransaction/get_signer_address-2	35.3ms ± 3%	37.4ms ±14%	~	(p=0.209 n=7+7)
RuntimeTransaction/get_public_account-2	38.8ms ± 2%	37.8ms ± 5%	~	(p=0.310 n=6+6)
RuntimeTransaction/get_account_and_get_balance-2	324ms ± 5%	327ms ± 3%	~	(p=0.318 n=7+7)
RuntimeTransaction/get_account_and_get_available_balance-2	289ms ± 6%	291ms ± 6%	~	(p=0.710 n=7+7)
RuntimeTransaction/get_account_and_get_storage_used-2	40.7ms ± 7%	39.8ms ± 4%	~	(p=0.209 n=7+7)
RuntimeTransaction/get_account_and_get_storage_capacity-2	267ms ± 3%	266ms ± 2%	~	(p=0.805 n=7+7)
RuntimeTransaction/get_signer_vault-2	43.7ms ± 8%	42.8ms ± 8%	~	(p=0.318 n=7+7)
RuntimeTransaction/get_signer_receiver-2	57.9ms ± 3%	56.6ms ± 5%	~	(p=0.097 n=7+7)
RuntimeTransaction/transfer_tokens-2	223ms ± 3%	225ms ± 2%	~	(p=0.620 n=7+7)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2	41.9ms ± 5%	42.1ms ± 2%	~	(p=0.259 n=7+7)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2	86.2ms ± 7%	88.1ms ± 4%	~	(p=0.318 n=7+7)
RuntimeTransaction/create_new_account-2	906ms ± 1%	910ms ± 3%	~	(p=0.318 n=7+7)
RuntimeTransaction/call_empty_contract_function-2	38.4ms ± 3%	38.4ms ± 4%	~	(p=0.902 n=7+7)
RuntimeTransaction/emit_event-2	53.5ms ± 4%	52.9ms ± 7%	~	(p=0.456 n=7+7)
RuntimeTransaction/borrow_array_from_storage-2	151ms ± 5%	148ms ± 2%	~	(p=0.209 n=7+7)
RuntimeTransaction/copy_array_from_storage-2	155ms ± 1%	155ms ± 3%	~	(p=0.836 n=6+7)
RuntimeNFTBatchTransfer-2	127ms ± 3%	128ms ± 4%	~	(p=0.535 n=7+7)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-2	5.00s ± 3%	4.93s ± 2%	~	(p=0.318 n=7+7)
	computation		delta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-2	202 ± 0%	202 ± 0%	~	(all equal)
RuntimeTransaction/convert_int_to_string-2	402 ± 0%	402 ± 0%	~	(all equal)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2	502 ± 0%	502 ± 0%	~	(all equal)
RuntimeTransaction/get_signer_address-2	302 ± 0%	302 ± 0%	~	(all equal)
RuntimeTransaction/get_public_account-2	402 ± 0%	402 ± 0%	~	(all equal)
RuntimeTransaction/get_account_and_get_balance-2	1.00k ± 0%	1.00k ± 0%	~	(all equal)
RuntimeTransaction/get_account_and_get_available_balance-2	2.60k ± 0%	2.60k ± 0%	~	(all equal)
RuntimeTransaction/get_account_and_get_storage_used-2	402 ± 0%	402 ± 0%	~	(all equal)
RuntimeTransaction/get_account_and_get_storage_capacity-2	1.70k ± 0%	1.70k ± 0%	~	(all equal)
RuntimeTransaction/get_signer_vault-2	402 ± 0%	402 ± 0%	~	(all equal)
RuntimeTransaction/get_signer_receiver-2	602 ± 0%	602 ± 0%	~	(all equal)
RuntimeTransaction/transfer_tokens-2	3.50k ± 0%	3.50k ± 0%	~	(all equal)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2	602 ± 0%	602 ± 0%	~	(all equal)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2	602 ± 0%	602 ± 0%	~	(all equal)
RuntimeTransaction/create_new_account-2	202 ± 0%	202 ± 0%	~	(all equal)
RuntimeTransaction/call_empty_contract_function-2	402 ± 0%	402 ± 0%	~	(all equal)
RuntimeTransaction/emit_event-2	602 ± 0%	602 ± 0%	~	(all equal)
RuntimeTransaction/borrow_array_from_storage-2	2.60k ± 0%	2.60k ± 0%	~	(all equal)
RuntimeTransaction/copy_array_from_storage-2	2.60k ± 0%	2.60k ± 0%	~	(all equal)
	interactions		delta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-2	212k ± 0%	212k ± 0%	~	(all equal)
RuntimeTransaction/convert_int_to_string-2	212k ± 0%	212k ± 0%	~	(all equal)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2	212k ± 0%	212k ± 0%	~	(all equal)
RuntimeTransaction/get_signer_address-2	212k ± 0%	212k ± 0%	~	(all equal)
RuntimeTransaction/get_public_account-2	212k ± 0%	212k ± 0%	~	(all equal)
RuntimeTransaction/get_account_and_get_balance-2	23.8M ± 0%	23.8M ± 0%	~	(all equal)
RuntimeTransaction/get_account_and_get_available_balance-2	5.43M ± 0%	5.43M ± 0%	~	(all equal)
RuntimeTransaction/get_account_and_get_storage_used-2	212k ± 0%	212k ± 0%	~	(all equal)
RuntimeTransaction/get_account_and_get_storage_capacity-2	5.43M ± 0%	5.43M ± 0%	~	(all equal)
RuntimeTransaction/get_signer_vault-2	212k ± 0%	212k ± 0%	~	(all equal)
RuntimeTransaction/get_signer_receiver-2	212k ± 0%	212k ± 0%	~	(all equal)
RuntimeTransaction/transfer_tokens-2	212k ± 0%	212k ± 0%	~	(all equal)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2	212k ± 0%	212k ± 0%	~	(all equal)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2	215k ± 0%	215k ± 0%	~	(all equal)
RuntimeTransaction/create_new_account-2	12.0M ± 0%	12.0M ± 0%	~	(all equal)
RuntimeTransaction/call_empty_contract_function-2	213k ± 0%	213k ± 0%	~	(all equal)
RuntimeTransaction/emit_event-2	213k ± 0%	213k ± 0%	~	(all equal)
RuntimeTransaction/borrow_array_from_storage-2	218k ± 0%	218k ± 0%	~	(all equal)
RuntimeTransaction/copy_array_from_storage-2	218k ± 0%	218k ± 0%	~	(all equal)
	alloc/op		delta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-2	33.9MB ± 4%	34.6MB ± 3%	~	(p=0.128 n=7+7)
RuntimeTransaction/convert_int_to_string-2	35.5MB ± 4%	35.2MB ± 4%	~	(p=0.535 n=7+7)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2	35.5MB ± 5%	35.3MB ± 6%	~	(p=1.000 n=7+7)
RuntimeTransaction/get_signer_address-2	34.7MB ± 3%	34.6MB ± 1%	~	(p=0.535 n=7+7)
RuntimeTransaction/get_public_account-2	36.2MB ± 5%	35.7MB ± 5%	~	(p=0.318 n=7+7)
RuntimeTransaction/get_account_and_get_balance-2	122MB ± 2%	124MB ± 1%	~	(p=0.209 n=7+7)
RuntimeTransaction/get_account_and_get_available_balance-2	105MB ± 4%	104MB ± 3%	~	(p=0.535 n=7+7)
RuntimeTransaction/get_account_and_get_storage_used-2	36.6MB ± 5%	36.0MB ± 3%	~	(p=0.318 n=7+7)
RuntimeTransaction/get_account_and_get_storage_capacity-2	103MB ± 3%	103MB ± 2%	~	(p=0.620 n=7+7)
RuntimeTransaction/get_signer_vault-2	36.9MB ± 7%	36.7MB ± 6%	~	(p=0.902 n=7+7)
RuntimeTransaction/transfer_tokens-2	84.1MB ± 4%	83.3MB ± 3%	~	(p=0.383 n=7+7)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2	36.1MB ± 4%	35.6MB ± 1%	~	(p=0.234 n=7+6)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2	52.2MB ± 6%	53.0MB ± 3%	~	(p=0.209 n=7+7)
RuntimeTransaction/create_new_account-2	186MB ± 3%	187MB ± 3%	~	(p=0.535 n=7+7)
RuntimeTransaction/call_empty_contract_function-2	35.8MB ± 5%	35.8MB ± 2%	~	(p=0.710 n=7+7)
RuntimeTransaction/emit_event-2	40.7MB ± 3%	39.7MB ± 5%	~	(p=0.165 n=7+7)
RuntimeTransaction/borrow_array_from_storage-2	69.6MB ± 2%	68.7MB ± 4%	~	(p=0.383 n=7+7)
RuntimeTransaction/copy_array_from_storage-2	82.0MB ± 3%	82.2MB ± 3%	~	(p=0.902 n=7+7)
RuntimeNFTBatchTransfer-2	53.8MB ± 4%	54.1MB ± 6%	~	(p=0.620 n=7+7)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-2	1.20GB ± 1%	1.20GB ± 1%	~	(p=1.000 n=7+7)
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/get_signer_receiver-2	42.4MB ± 3%	41.2MB ± 3%	−2.96%	(p=0.017 n=7+7)
	allocs/op		delta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-2	80.0k ± 0%	80.0k ± 0%	+0.01%	(p=0.042 n=7+7)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2	194k ± 0%	194k ± 0%	+0.01%	(p=0.028 n=6+7)
RuntimeTransaction/convert_int_to_string-2	92.2k ± 0%	92.2k ± 0%	~	(p=0.804 n=7+7)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2	103k ± 0%	103k ± 0%	~	(p=0.268 n=7+5)
RuntimeTransaction/get_signer_address-2	84.2k ± 0%	84.2k ± 0%	~	(p=0.689 n=7+7)
RuntimeTransaction/get_public_account-2	106k ± 0%	106k ± 0%	~	(p=0.882 n=7+7)
RuntimeTransaction/get_account_and_get_balance-2	1.32M ± 0%	1.32M ± 0%	~	(p=0.710 n=7+7)
RuntimeTransaction/get_account_and_get_available_balance-2	1.20M ± 0%	1.20M ± 0%	~	(p=0.596 n=7+7)
RuntimeTransaction/get_account_and_get_storage_used-2	116k ± 0%	116k ± 0%	~	(p=0.917 n=7+7)
RuntimeTransaction/get_account_and_get_storage_capacity-2	1.12M ± 0%	1.12M ± 0%	~	(p=0.735 n=7+7)
RuntimeTransaction/get_signer_vault-2	121k ± 0%	121k ± 0%	~	(p=0.645 n=7+7)
RuntimeTransaction/transfer_tokens-2	840k ± 0%	839k ± 0%	~	(p=0.209 n=7+7)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2	119k ± 0%	119k ± 0%	~	(p=0.432 n=7+6)
RuntimeTransaction/create_new_account-2	2.31M ± 0%	2.31M ± 0%	~	(p=1.000 n=7+7)
RuntimeTransaction/call_empty_contract_function-2	95.6k ± 0%	95.6k ± 0%	~	(p=0.471 n=7+7)
RuntimeTransaction/emit_event-2	135k ± 0%	135k ± 0%	~	(p=0.734 n=7+7)
RuntimeTransaction/borrow_array_from_storage-2	334k ± 0%	334k ± 0%	~	(p=0.387 n=6+7)
RuntimeTransaction/copy_array_from_storage-2	290k ± 0%	290k ± 0%	~	(p=0.646 n=7+7)
RuntimeNFTBatchTransfer-2	267k ± 1%	267k ± 1%	~	(p=1.000 n=7+7)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-2	16.9M ± 0%	16.9M ± 0%	~	(p=0.456 n=7+7)
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/get_signer_receiver-2	195k ± 0%	195k ± 0%	−0.01%	(p=0.034 n=7+6)
	us/tx		delta
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-2	2.44k ± 3%	2.40k ± 2%	~	(p=0.301 n=7+7)

[codecov-commenter]

Codecov Report

Merging #2647 (d7f6fcc) into master (69f0be7) will decrease coverage by 1.65%. The diff coverage is 67.19%.

@@            Coverage Diff             @@
##           master    #2647      +/-   ##
==========================================
- Coverage   55.50%   53.84%   -1.66%     
==========================================
  Files         765      617     -148     
  Lines       69798    53081   -16717     
==========================================
- Hits        38739    28584   -10155     
+ Misses      27881    22162    -5719     
+ Partials     3178     2335     -843     

Flag	Coverage Δ
unittests	53.84% <67.19%> (-1.66%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
cmd/scaffold.go	15.75% <0.00%> (-0.02%)	:arrow_down:
model/chunks/executionDataFaults.go	0.00% <0.00%> (ø)
model/flow/chunk.go	50.00% <ø> (ø)
network/p2p/p2pbuilder/libp2pNodeBuilder.go	0.00% <0.00%> (ø)
network/p2p/unicast/ratelimit/noop_rate_limiter.go	0.00% <0.00%> (ø)
network/p2p/unicast/ratelimit/rate_limiters.go	0.00% <0.00%> (ø)
network/validator/any_validator.go	0.00% <0.00%> (ø)
network/validator/not_validator.go	0.00% <0.00%> (ø)
network/validator/origin_validator.go	0.00% <0.00%> (ø)
network/validator/sender_validator.go	0.00% <0.00%> (ø)
... and 181 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[peterargue]

closing in favor of new refactored version: https://github.com/onflow/flow-go/pull/4586