SSE subscription for newPayload event to trigger block building

[metachris]

Block builders in the MEV-Boost ecosystem need a few bits of information to start block building as soon as possible, which is (a) at the newPayload event, or (b) after a timeout if the BN missed a slot, so block production is still triggered.

Currently a custom BN fork is needed to trigger block production in time.

We'd like to propose adding another SSE event subscription to the beacon-APIs, which can be used to trigger block production.

The necessary payload fields are slot, blockHash and prevRandao (current implementation here)

I can open a PR with the changes to https://github.com/ethereum/beacon-APIs/blob/master/apis/eventstream/index.yaml but wanted to open the issue to discuss this first.

---

For reference, this is currently implemented in a custom BN fork with a notifyBuildBlock method which send BuilderPayloadAttributes, and is triggered on onBlock and other events and after a timeout has been reached (see process_block.go)

Edit 1: should the event be sent again if the likely parent changes due to new attestations?

[ralexstokes]

I think if we add something like this it should specifically support the use case where a builder builds on top of many heads in parallel (so all this event stream does is give you the stream of validated new execution heads, and it is the builder's responsibility to track the execution block tree)

Builders can consume a separate stream with the existing head event to know the head of the canonical chain -- and if builders want more sophisticated fork choice information I feel at the moment it is fine to expect custom implementation. Happy to hear different views here.

[avalonche]

The current custom BN fork sends the block building trigger for every valid payload received. This will support the use case of building on top of many heads in parallel.

[terencechain]

I agree with @ralexstokes , it's useful for builders to build on top of many heads in parallel. Why not just add a payload_block_hash field to the head event

[metachris]

Some notes:

1. the necessary additional information is block_hash and prev_randao
2. the start-block-building event is needed also in case of missed slots
3. would the head event fire multiple times in case of multiple possible parents?

For reference, the head event currently has this data:

{
    "slot": "4774249",
    "block": "0xaa979a3af687d541bf45819c769e2c1ffcbbb4c5300ef940f8ee28ee004f2e86",
    "state": "0x851e4fb018a126eea86e8aea9f7291253e4f172fd6e864ae3a579660ad2d823b",
    "epoch_transition": false,
    "execution_optimistic": false,
    "previous_duty_dependent_root": "0x76c97e074bf5acec4535f3b5eafe54ba773edbad4edd5109a5089ac90b9e77f6",
    "current_duty_dependent_root": "0x9ae869915da85e19d48dd0ca71582e2b80fd90b52fbb888156eec19555da734b"
}

[terencechain]

1. Yeah. That's what I meant for payload_block_hash, we could add prev_randao as well

2. relayer/builder can monitor block SSE and see if there's no new block in the 4s mark, then start building via the previous head

3. It'll file when there's a new head which should be sufficient

[michaelsproul]

Now that withdrawals are also in the mix I think it makes less sense to fire this event for the builder on each newPayload call. The reason is that the expected withdrawals can change depending on the slot that the new block will be proposed at (if an epoch transition happens it can change balances, which can change the withdrawals). So the CL would need to re-publish new events at each skipped slot (as alluded to in the OP). There's also a bit of work that the CL needs to do to compute the withdrawals, we may need to run an epoch transition on the state in order to get the withdrawals, and depending on the client architecture this may or may not be happening on newPayload. For example, in Lighthouse we do this when computing payloadAttributes for forkchoiceUpdated.

The alternative I'd like to propose is that CL clients publish augmented payload attributes via the SSE stream, whenever they "discover" new payload attributes (which can be client specific).

The event I'd propose is:

{
    "proposer_index": "123",
    "proposal_slot": "54321",
    "proposal_block_number": "654321",
    "parent_block_root": "0xff...",
    "parent_block_hash": "0xff..",
    "payload_attributes": {
        "timestamp": "12345",
        "prev_randao": "0xff..",
        "suggested_fee_recipient": "0xff..",
        "withdrawals": [],
    },
    "withdrawals_root": "0xff..",
    "version": "capella"
}

The withdrawals_root is maybe a bit unnecessary. Relays and builders probably have the tools to compute the withdrawals root (SSZ) or hash (RLP) without the CL's help.

[Ruteri]

@michaelsproul This proposal fits the builder use case perfectly.

CL clients publish augmented payload attributes via the SSE stream, whenever they "discover" new payload attributes

As I understand it also covers the case of missed slots, which is always a headache.

[michaelsproul]

Work in progress implementation for Lighthouse is here: https://github.com/sigp/lighthouse/pull/4027. I've omitted the withdrawals_root for now but could easily add it. Also added a version tag as per @mcdee's excellent suggestion.

I will open a PR here with a spec in the next few days as well.

[ralexstokes]

@michaelsproul have you thought about having this event fire for each head a node is aware of?

or do we want to say this endpoint only services the canonical head?

[michaelsproul]

@ralexstokes I did briefly consider that, but it's substantially more complicated to implement. We already have all the machinery for computing payload attributes from the canonical head (or its parent in the case of a 1-slot re-org), but would have to separately iterate the other heads, load their BeaconStates (expensive) and compute the withdrawals.

That said, I think the proposed API would be backwards compatible with this strategy, so we could add it in future.

I can imagine builders running a few different CLs with different configurations and combining the streams to achieve better coverage on attributes. E.g. Lighthouse could be run with and without proposer re-orgs enabled.