[DataAvailability] Improve network reliability by reducing API load on execution node

[peterargue]

Objective: Improve network reliability by reducing API load on the execution node

KR1: Accelerate execution data availability on the access node

Access nodes fetch execution data from execution nodes over the peer-to-peer libp2p network. Currently, execution data for a block is fetched by the access node only after the block has been sealed. This means that Access API requests that require execution data for a block that has not yet been sealed are forwarded to execution nodes until the block is sealed. Such API requests include the following:

1. Script execution on a finalized or executed block that has not yet been sealed.
2. Streaming API for Transaction status.

This feature makes execution data for a block available on the access node soon after the block has been executed. This helps achieve the following:

1. Reduce the API calls forwarded to the execution node by the access node and thus reduce the load on the execution node.
2. In the future it allows for the execution node to completely shut off its gRPC API endpoint and only be reachable via the peer-to-peer network.

DACI

Driver	Approver	Consulted	Informed
KROK team	@peterargue	Flow Team	Flow Team, marketing

How we measure success

• Execution data-related API calls are reduced by x% on average on an execution node.
• BlockExecutionData is downloaded by Access nodes before the block is sealed. Ideally as soon after the block was executed as possible.
• Data from orphaned forks is pruned from the db.
• There are metrics and logging that provide observibility into how the system is performing, and the latency.
• There is unit and integration test coverage for the new features.
• There is documentation both in code and a readme outlining how the system works.

---

KR2: Support soft finality for EVM Transactions

Currently, EVM GW returns a Transaction Receipt for a transaction only after the transaction has been sealed. However, this precludes a dApp from relying on soft-finality which in Flow means state after the transaction has been executed but not yet sealed. The Time To Execution is currently at ~4 seconds (50th percentile) while Time to Seal is at 14 seconds. If dApps, which are comfortable with soft-finality, could query for state after 4 seconds instead of waiting for 14 seconds, it could result in a significant improvement user experience.

This KR entails the following:

1. EVM GW receiving transacition events from the Access node as soon as the transaction is executed (automatic consequence of KR1).
2. EVM GW being able to handle forks and discard events of blocks on a fork.

DACI

Driver	Approver	Consulted	Informed
KROK team	@peterargue	Flow Team	Flow Team, marketing

How we measure success

• EVM Gateway returns transaction receipt after a transaction has been executed but before its sealed.
• EVM Gateway can serve state queries after the transaction has been executed but before its sealed reflecting the state change due to the transaction.

Detail Technical Problem Definition

Currently, the ExecutionDataRequester waits until a block is sealed before it begins to download its BlockExecutionData. This simplified the initial implementation at the expense of added latency. Now that execution data is indexed and the data is used to serve the Access API, there's greater need to reduce that latency. A few of the main reasons:

• Currently about 30% of script executions on mainnet are for blocks that have not yet been indexed by the AN. Reducing the latency would allow ANs to serve an even higher percent.
• As data sizes get larger (more tx/events, larger account storage, etc) it will take longer to download the data. Starting earlier will ensure it's available within a reasonable amount of time.
• Some users want access to data as soon as the block is executed. By syncing before sealed, the data will become available several seconds sooner and likely within a tolerable range for these users.
• Eventually, we will want to phase out the use of Execution nodes to service requests from Access nodes for the Access API.

Proposed Solution

Update the requester to sync execution data for blocks that are executed but not yet sealed. There is an initial implementation that we can use as a starting point: https://github.com/onflow/flow-go/pull/2842. This may also work as-is. We originally did not deploy this to keep the system simpler during it's initial rollout.

Staked access nodes subscribe to the ExecutionReceipts channel which is where Execution nodes publish notifications for newly executed blocks. These block may or may not be finalized yet.

The original issue description is here

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[AlexHentschel]

I am wondering if we should embed this issue into a broader user story. I like the present issue, as it focuses on the central technical features that have to be delivered. Though, there are adjacent aspects that are important from my perspective for this work to fully unfold its ecosystem impact (not sure if we already have additional issues, so I am putting my thoughts here):

• it would be good if clients could query transaction results already when there is "soft finality" (some EN - similarly to an L2 - has made a commitment to the result, and will be slashed if that result turns out to be incorrect).
  • it should be up to the client whose execution nodes they are willing to trust (most convenient and versatile would be if the client could specify "I am happy to take a result the result was committed to by any one of the ENs in the set {...}"
  • The client explicitly accepts the risk that they get a wrong result if one of their specified ENs happens to be byzantine
  • The protocol will eventually slash the byzantine EN(s) and orphan the incorrect result (wrong results are guaranteed to not be sealed). The client must be able to do the same and recover itself from a result that they chose to accept but eventually turned out to be wrong (if the client implementation can't resolve this situation it will be stuck or has to always wait for sealing)
• This ☝ has to be transparently documented in our developer docs. It needs to be worded well and explained well to not scare developers. We need to clearly communicate that people who are happy with optimistic L2s (or centralized L2s like base), should not be scared about using flow's soft finality (essentially they get the same guarantees, if not even stronger guarantees on flow compared to optimistic/centralized L2s). Preferably, developer-facing documenting and positioning of flow's soft finality [TTE] should be driven by the DevX and marketing teams, with protocol contributing on the technical aspects.
• marketing campaign to draw attention
• get our metrics on Chainspect updated (👉slack marketing channel)

[bluesign]

First of all I love this, this is really huge problem now, you send the transaction, approx 4-5 seconds you get the result with executed status, but then it tools like 5-6 seconds more to scripts to catch up. ( so you need to much handling to make good UX )

I think this should be an option in FCL and other SDKs ( can default to sealed but user can change to soft finality etc )

Not sure if it will help with Chainspect stats though, from I read ( saw the site first time ) they seem not to use soft finality.

Time To Finality (TTF) is the duration of time it takes for a transaction to be considered final in a blockchain network. It’s essentially the time it takes for a transaction to become permanent in a blockchain’s ledger.

From this I understand; Technically either without this change also it should be changed to soft finality on Chainspect or it will stay like this after this change too.

[vishalchangrani]

Which Access APIs will this change effect? I agree with Alex on making this a broader user story and including fcl related changes as part of the KR as well similar to what we are doing for websocket streaming.

[vishalchangrani]

reworded the OKR, added DACI.

[franklywatson]

PoC of interim version of soft finality was successfully deployed on preview for initial testing. Assuming that things go well and one any issues are addressed we will be putting it on testnet towards the latter half of this week

[vishalchangrani]

Given the HCUs and P0 issues, the production version of the soft-finality has been pushed out by several weeks.

[vishalchangrani]

Design doc for Execution data on AN has been published by Peter

[vishalchangrani]

Use story on Dev-rel to measure the API improvements through some of the real-world apps on Flow EVM - https://github.com/onflow/dev-rel/issues/66

[vishalchangrani]

Issue for production ready soft-finality on EVM GW - https://github.com/onflow/flow-evm-gateway/issues/756

[j1010001]

KR1 milestone breakdown: https://www.notion.so/flowfoundation/Optimistic-Syncing-System-Design-1c11aee12324807d9b42f3f25f6ab7db?pvs=4#1dd1aee1232480c9a731ff386d57c97d

[vishalchangrani]

Milestone 1 of optimistic sync in close to completion. However, this OKR will continue in the next cycle.

[vishalchangrani]

Krok team update:

1. Working on PR comments for previous issues.
2. Started working on new issues from Milestone 2 of optimistic syncing.

[vishalchangrani]

What are the three milestones? Milestone 1 - Build the processing pipeline engine Milestone 2 - Integrate pipeline into a fork-aware system Milestone 3 - API changes.

[vishalchangrani]

Milestone 1 is almost complete. 3 issues in review 1 issue in development (functional testing)

Milestone 2 is in progress Continuing work on the results forest Starting ground work for refactoring ingestion Starting design for storage layer

[vishalchangrani]

1. Milestone 1 is almost complete.

• 2 issues in review
• 1 issue in development (functional testing)

2. Milestone 2 is in progress

• Continuing work on the results forest
• Initial ingestion engine refactoring in progress (1 merged, 1 in review, 1 in progress)
• Initial draft of caching layer is complete. leo did initial review before he left

[vishalchangrani]

Milestone 1 - we've finished all of the development for milestone 1. only remaining task is function testing which is in review.

Milestone 2 - In progress. Yurii is starting to get up to speed on the project to help with design and code reviews.

Milestone 3 - KROK is getting started on the API changes. doing some initial refactoring PRs right now.

[vishalchangrani]

• Milestone 1 is still waiting on that testing PR. it's blocked on some changes Yurii and Peter are working on
• Milestone 2 is continuing to make progress, no major updates
• Milestone 3 has 3 issues in progress, none completed.

[vishalchangrani]

• milestone 1 is still waiting on that testing PR. it's now unblocked and in-progress
• milestone 2 is continuing to make progress, no major updates
• milestone 3 has 4 issues in progress, 1 completed.

[vishalchangrani]

all three milestones in progress.

[vishalchangrani]

• milestone 1 is now complete
• milestone 2: Epic 7180 and Epic 7181 are in progress (4/11 and 1/1 - total number of issues will go up)
• milestone 3: Epic 7182 and Epic 7610 are in progress (6/14 and 2/7)

[vishalchangrani]

• milestone 1 is now complete
• milestone 2: Epic 7180 and Epic 7181 are in progress (4/11 and 1/1)
• milestone 3: Epic 7182, Epic 7610, Epic 7615 are in progress (6/14, 2/7, 2 in progress/12),

[vishalchangrani]

• milestone 1 is now complete
• milestone 2: Epic 7180 and Epic 7181 are in progress (4/11 and 1/1)
• milestone 3: Epic 7182, Epic 7610, Epic 7615 are in progress (6/14, 2/7, 0/12)
  • Current focus: Epic 7615: 5 in review and 1 in progress out of 12. In Review: [Data Availability] Implement fork-aware Events Endpoints #7652 [Data Availability] Implement fork-aware Streaming Events Endpoints #7657 [Data Availability] Implement fork-aware Streaming Account Events Endpoints #7658 [Data Availability] Implement fork-aware Transaction Results Endpoints #7644 [Data Availability] Implement fork-aware System Transaction Endpoints #7648 In Progress: [Data Availability] Implement fork-aware Account Endpoints #7650

[vishalchangrani]

• milestone 1: completed
• milestone 2: - Epic 7180: in progress: 4 done, 5 in progress out of 11 - Epic 7181: 1 done, more issues to be created after 7180
• milestone 3: - Epic 7182: 6/14 done, 2 in progress - Epic 7615: 1 in progress, 6 in review, out of 12 - Epic 7610: 2/7 done, 0 in progress

[vishalchangrani]

• milestone 1: completed
• milestone 2: - Epic 7180: 5 out of 12 done - Epic 7181: 1 out 1 done, more issues to be created after 7180
• milestone 3: - Epic 7182: 6/17 done - Epic 7615: 10/12 done - Epic 7610: 2/7 done

[vishalchangrani]

In Review: [Data Availability] Implement fork-aware Events Endpoints #7652 [DataAvailability] refactor events test in http package #7923

In Progress: [Data Availability] Implement fork-aware Streaming Account Events Endpoints #7658 [Data Availability] Implement fork-aware Streaming Events Endpoints #7657 [Data Availability] Implement fork-aware Script Endpoints #7651 [DataAvailability] Implement fork-aware Execution Data Endpoints #7656

[vishalchangrani]

Milestones status:

• milestone 1: completed
• milestone 2:
  • https://github.com/onflow/flow-go/issues/7180: 4 done, 3 in progress out of 12
  • https://github.com/onflow/flow-go/issues/7181: 1 done, more issues to be created after 7180
• milestone 3:
  • https://github.com/onflow/flow-go/issues/7182: 6/17 done ( some new sub-issues added )
  • https://github.com/onflow/flow-go/issues/7615: 5 in progress, 3 in review, out of 12
  • https://github.com/onflow/flow-go/issues/7610: 2/7 done, 0 in progress

[vishalchangrani]

Milestones status:

• milestone 1: completed
• milestone 2: - Epic 7180: 4 done, 3 in progress out of 12 - Epic 7181: 1 done, more issues to be created after 7180
• milestone 3: - Epic 7182: 6/17 done ( some new sub-issues added ) - Epic 7615: 5 in progress, 3 in review, out of 12 - Epic 7610: 2/7 done, 0 in progress