flow-go icon indicating copy to clipboard operation
flow-go copied to clipboard

Published 20 hours ago • verified publisher icon onflow

Emit Protocol Events after database transaction

Open jordanschalm opened this issue 2 years ago • 2 comments

This PR modifies protocol events:

  • protocol events are emitted after the corresponding database transaction completes successfully
    • protocol events may not be delivered, only due to a crash/restart, and only if the corresponding state change (eg. block 100 is finalized) is readable in the Protocol State when the process restarts
  • documentation is updated to describe delivery guarantees of protocol events
  • adds documentation to all protocol event consumers describing how dropped protocol events due to crashes are handled
  • updates ReactorEngine to handle missed EpochCommittedPhaseStarted events (this was the only case where the kind of possible dropped events were not already handled)

More detail in https://github.com/dapperlabs/flow-go/issues/6317

jordanschalm avatar Aug 16 '22 18:08 jordanschalm

Codecov Report

Merging #2994 (1467c48) into feature/active-pacemaker (fd71d16) will decrease coverage by 0.61%. The diff coverage is 57.50%.

@@                     Coverage Diff                      @@
##           feature/active-pacemaker    #2994      +/-   ##
============================================================
- Coverage                     55.37%   54.75%   -0.62%     
============================================================
  Files                           702      688      -14     
  Lines                         66859    64579    -2280     
============================================================
- Hits                          37020    35361    -1659     
+ Misses                        26787    26321     -466     
+ Partials                       3052     2897     -155
Flag Coverage Δ
unittests 54.75% <57.50%> (-0.62%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
engine/execution/ingestion/engine.go 52.32% <ø> (ø)
storage/badger/operation/dkg.go 78.26% <ø> (ø)
engine/consensus/dkg/reactor_engine.go 62.88% <28.57%> (-1.61%) :arrow_down:
engine/collection/epochmgr/engine.go 73.77% <83.33%> (-0.46%) :arrow_down:
state/protocol/badger/mutator.go 65.80% <100.00%> (-0.06%) :arrow_down:
...sus/approvals/assignment_collector_statemachine.go 42.75% <0.00%> (-5.80%) :arrow_down:
fvm/environment/contract_updater.go 66.54% <0.00%> (ø)
...s/hotstuff/signature/randombeacon_reconstructor.go
consensus/hotstuff/verification/staking_signer.go
...tstuff/votecollector/combined_vote_processor_v3.go
... and 105 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

codecov-commenter avatar Aug 17 '22 00:08 codecov-commenter

FVM Benchstat comparison

This branch with compared with the base branch onflow:feature/active-pacemaker commit fd71d161aad0331043b64a07b2fcd5ae29f0683a

The command (for i in {1..10}; do go test ./fvm ./engine/execution/computation --bench . --tags relic -shuffle=on --benchmem --run ^$; done) was used.

Collapsed results for better readability

old.txtnew.txt
time/opdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/call_empty_contract_function-228.6ms ± 8%30.5ms ± 5%+6.84%(p=0.001 n=10+10)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-229.8ms ± 6%31.3ms ± 7%+5.00%(p=0.035 n=9+10)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-24.72s ± 1%4.90s ± 3%+3.74%(p=0.000 n=8+10)
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeNFTBatchTransfer-2115ms ± 3%118ms ± 5%+3.22%(p=0.004 n=9+9)
RuntimeTransaction/copy_array_from_storage-2129ms ± 3%133ms ± 4%+3.11%(p=0.002 n=10+10)
RuntimeTransaction/get_account_and_get_balance-2263ms ± 3%271ms ± 3%+3.08%(p=0.001 n=10+9)
RuntimeTransaction/emit_event-243.6ms ± 5%44.9ms ± 5%+3.00%(p=0.019 n=10+10)
RuntimeTransaction/get_signer_receiver-246.2ms ± 6%47.5ms ± 3%+2.81%(p=0.043 n=10+10)
RuntimeTransaction/create_new_account-2853ms ± 2%875ms ± 4%+2.55%(p=0.011 n=10+10)
RuntimeTransaction/get_account_and_get_storage_capacity-2211ms ± 2%216ms ± 1%+2.13%(p=0.000 n=9+7)
RuntimeTransaction/borrow_array_from_storage-2130ms ± 4%133ms ± 4%+2.08%(p=0.029 n=10+10)
RuntimeTransaction/transfer_tokens-2205ms ± 1%207ms ± 3%+1.36%(p=0.019 n=9+9)
RuntimeTransaction/reference_tx-225.4ms ±15%26.0ms ± 9%~(p=0.247 n=10+10)
RuntimeTransaction/convert_int_to_string-227.3ms ± 9%28.5ms ± 8%~(p=0.079 n=9+10)
RuntimeTransaction/get_signer_address-226.5ms ± 6%26.0ms ±11%~(p=0.280 n=10+10)
RuntimeTransaction/get_public_account-230.6ms ±13%30.4ms ±14%~(p=0.912 n=10+10)
RuntimeTransaction/get_account_and_get_available_balance-2249ms ± 7%244ms ± 3%~(p=0.481 n=10+10)
RuntimeTransaction/get_account_and_get_storage_used-232.8ms ± 5%33.5ms ± 6%~(p=0.190 n=10+10)
RuntimeTransaction/get_signer_vault-235.6ms ± 5%35.4ms ±10%~(p=0.853 n=10+10)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-234.0ms ± 5%34.9ms ± 7%~(p=0.105 n=10+10)
RuntimeTransaction/load_and_save_long_string_on_signers_address-277.9ms ± 1%78.5ms ± 5%~(p=0.315 n=8+10)
 
alloc/opdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeNFTBatchTransfer-259.0MB ± 5%58.7MB ± 3%~(p=1.000 n=10+10)
RuntimeTransaction/reference_tx-236.2MB ± 6%37.0MB ± 5%~(p=0.218 n=10+10)
RuntimeTransaction/convert_int_to_string-237.6MB ± 7%38.4MB ± 8%~(p=0.280 n=10+10)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-239.3MB ± 5%38.9MB ± 6%~(p=0.436 n=10+10)
RuntimeTransaction/get_signer_address-237.5MB ± 5%36.3MB ± 9%~(p=0.133 n=9+10)
RuntimeTransaction/get_public_account-239.7MB ± 2%38.6MB ± 7%~(p=0.237 n=8+10)
RuntimeTransaction/get_account_and_get_storage_used-240.0MB ± 4%39.8MB ± 6%~(p=0.912 n=10+10)
RuntimeTransaction/get_account_and_get_storage_capacity-2107MB ± 2%106MB ± 4%~(p=0.353 n=10+10)
RuntimeTransaction/get_signer_vault-241.0MB ± 3%39.8MB ± 8%~(p=0.278 n=9+10)
RuntimeTransaction/get_signer_receiver-243.5MB ± 4%44.0MB ± 6%~(p=0.353 n=10+10)
RuntimeTransaction/transfer_tokens-288.1MB ± 2%89.0MB ± 3%~(p=0.190 n=10+10)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-238.7MB ± 7%39.4MB ± 9%~(p=0.280 n=10+10)
RuntimeTransaction/load_and_save_long_string_on_signers_address-256.5MB ± 2%56.6MB ± 4%~(p=0.684 n=10+10)
RuntimeTransaction/create_new_account-2202MB ± 3%204MB ± 2%~(p=0.481 n=10+10)
RuntimeTransaction/call_empty_contract_function-237.8MB ± 8%39.0MB ± 5%~(p=0.143 n=10+10)
RuntimeTransaction/emit_event-242.1MB ± 4%42.2MB ± 5%~(p=0.739 n=10+10)
RuntimeTransaction/borrow_array_from_storage-269.9MB ± 5%69.8MB ± 4%~(p=0.971 n=10+10)
RuntimeTransaction/copy_array_from_storage-282.2MB ± 5%82.9MB ± 3%~(p=0.218 n=10+10)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-21.34GB ± 1%1.34GB ± 1%~(p=0.579 n=10+10)
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/get_account_and_get_balance-2130MB ± 2%128MB ± 3%−1.89%(p=0.007 n=10+10)
RuntimeTransaction/get_account_and_get_available_balance-2113MB ± 2%110MB ± 4%−2.46%(p=0.011 n=10+10)
 
allocs/opdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeNFTBatchTransfer-2293k ± 1%292k ± 1%~(p=0.542 n=10+10)
RuntimeTransaction/reference_tx-279.0k ± 0%79.0k ± 0%~(p=0.670 n=10+10)
RuntimeTransaction/convert_int_to_string-293.2k ± 0%93.2k ± 0%~(p=0.811 n=10+10)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2107k ± 0%107k ± 0%~(p=0.515 n=10+10)
RuntimeTransaction/get_signer_address-284.2k ± 0%84.2k ± 0%~(p=0.868 n=10+10)
RuntimeTransaction/get_public_account-2107k ± 0%107k ± 0%~(p=0.566 n=10+10)
RuntimeTransaction/get_account_and_get_balance-21.51M ± 0%1.51M ± 0%~(p=0.089 n=10+10)
RuntimeTransaction/get_account_and_get_available_balance-21.40M ± 0%1.40M ± 0%~(p=0.271 n=10+10)
RuntimeTransaction/get_account_and_get_storage_used-2128k ± 0%128k ± 0%~(p=0.839 n=10+10)
RuntimeTransaction/get_account_and_get_storage_capacity-21.24M ± 0%1.24M ± 0%~(p=0.853 n=10+10)
RuntimeTransaction/get_signer_vault-2127k ± 0%127k ± 0%~(p=0.897 n=10+10)
RuntimeTransaction/get_signer_receiver-2204k ± 0%204k ± 0%~(p=0.382 n=10+10)
RuntimeTransaction/transfer_tokens-2935k ± 0%935k ± 0%~(p=0.280 n=10+10)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2129k ± 0%129k ± 0%~(p=0.118 n=10+10)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2232k ± 0%232k ± 0%~(p=0.436 n=10+10)
RuntimeTransaction/create_new_account-22.67M ± 0%2.67M ± 0%~(p=0.172 n=10+10)
RuntimeTransaction/call_empty_contract_function-295.9k ± 0%95.9k ± 0%~(p=0.362 n=10+10)
RuntimeTransaction/emit_event-2136k ± 0%136k ± 0%~(p=0.541 n=10+10)
RuntimeTransaction/borrow_array_from_storage-2368k ± 0%368k ± 0%~(p=0.838 n=10+10)
RuntimeTransaction/copy_array_from_storage-2324k ± 0%324k ± 0%~(p=0.089 n=9+9)
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-220.5M ± 0%20.5M ± 0%~(p=0.912 n=10+10)
 
computationdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-2202 ± 0%202 ± 0%~(all equal)
RuntimeTransaction/convert_int_to_string-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-2502 ± 0%502 ± 0%~(all equal)
RuntimeTransaction/get_signer_address-2302 ± 0%302 ± 0%~(all equal)
RuntimeTransaction/get_public_account-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_balance-21.00k ± 0%1.00k ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_available_balance-22.60k ± 0%2.60k ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_storage_used-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_storage_capacity-21.30k ± 0%1.30k ± 0%~(all equal)
RuntimeTransaction/get_signer_vault-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/get_signer_receiver-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/transfer_tokens-23.50k ± 0%3.50k ± 0%~(all equal)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/load_and_save_long_string_on_signers_address-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/create_new_account-2202 ± 0%202 ± 0%~(all equal)
RuntimeTransaction/call_empty_contract_function-2402 ± 0%402 ± 0%~(all equal)
RuntimeTransaction/emit_event-2602 ± 0%602 ± 0%~(all equal)
RuntimeTransaction/borrow_array_from_storage-22.60k ± 0%2.60k ± 0%~(all equal)
RuntimeTransaction/copy_array_from_storage-22.60k ± 0%2.60k ± 0%~(all equal)
 
interactionsdelta
pkg:github.com/onflow/flow-go/fvm goos:linux goarch:amd64
RuntimeTransaction/reference_tx-244.3k ± 0%44.3k ± 0%~(all equal)
RuntimeTransaction/convert_int_to_string-244.3k ± 0%44.3k ± 0%~(all equal)
RuntimeTransaction/convert_int_to_string_and_concatenate_it-244.3k ± 0%44.3k ± 0%~(all equal)
RuntimeTransaction/get_signer_address-244.3k ± 0%44.3k ± 0%~(all equal)
RuntimeTransaction/get_public_account-244.3k ± 0%44.3k ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_balance-216.7M ± 0%16.7M ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_available_balance-25.13M ± 0%5.13M ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_storage_used-244.3k ± 0%44.3k ± 0%~(all equal)
RuntimeTransaction/get_account_and_get_storage_capacity-25.13M ± 0%5.13M ± 0%~(all equal)
RuntimeTransaction/get_signer_vault-244.5k ± 0%44.5k ± 0%~(all equal)
RuntimeTransaction/get_signer_receiver-244.9k ± 0%44.9k ± 0%~(all equal)
RuntimeTransaction/transfer_tokens-245.2k ± 0%45.2k ± 0%~(all equal)
RuntimeTransaction/load_and_save_empty_string_on_signers_address-244.4k ± 0%44.4k ± 0%~(all equal)
RuntimeTransaction/load_and_save_long_string_on_signers_address-247.2k ± 0%47.2k ± 0%~(all equal)
RuntimeTransaction/create_new_account-28.39M ± 0%8.39M ± 0%~(all equal)
RuntimeTransaction/call_empty_contract_function-244.5k ± 0%44.5k ± 0%~(all equal)
RuntimeTransaction/emit_event-244.5k ± 0%44.5k ± 0%~(all equal)
RuntimeTransaction/borrow_array_from_storage-249.7k ± 0%49.7k ± 0%~(all equal)
RuntimeTransaction/copy_array_from_storage-249.7k ± 0%49.7k ± 0%~(all equal)
 
us/txdelta
pkg:github.com/onflow/flow-go/engine/execution/computation goos:linux goarch:amd64
ComputeBlock/16/cols/128/txes-22.31k ± 1%2.39k ± 3%+3.74%(p=0.000 n=8+10)
 

github-actions[bot] avatar Sep 20 '22 14:09 github-actions[bot]

bors merge

jordanschalm avatar Oct 28 '22 07:10 jordanschalm

Build succeeded:

bors[bot] avatar Oct 28 '22 07:10 bors[bot]