ecosystem-contributions icon indicating copy to clipboard operation
ecosystem-contributions copied to clipboard

Published 20 hours ago • verified publisher icon ethereum-optimism

Foundation Mission Request: Leader Election Proof of Concept

Open bdresser opened this issue 1 year ago • 21 comments

Foundation Mission Request – Leader Election Proof of Concept

To take on this project, submit a proposal to this thread by June 28. Read more about Missions here.

  • S4 Intent: Progress Towards Technical Decentralization
  • Proposal Tier: Ember
  • Baseline grant amount: 30k OP
  • Accepting one or multiple proposals: Multiple
  • OP Labs or Optimism Foundation Sponsor: Mark (@tynesol)
  • Submit by: June 28th at 19:00 GMT
  • Selection by: July 13th at 19:00 GMT

Background

What is Leader Election?

Leader election is simply the ability to have different leaders that are responsible for creating the next canonical state transition in a distributed system. In a blockchain, leader election gives the ability for different block producers to produce blocks at different times. Leader election algorithms can be racy or race free.

A racy leader election algorithm is where many potential leaders are competing at the same time to be the leader, this is the case with proof of work. A race free leader election algorithm is when there is a single known leader at a particular point in time and there is no other way for a different potential leader to be the leader at that time, this is the case with Ethereum Gasper.

With Proposer-Builder Separation, the block builder (also known as a validator or a proposer) is virtualized in the sense where a network of block builders that are ultimately responsible for selecting the ordering of the transactions and then the proposer is simply responsible for signing off on the block. This turns the single entity that was responsible for producing a block at a particular slot into many possible entities that are all competing for building the most profitable block during the slot, so the raciness has returned because of MEV.

It is very difficult to understand the various second order effects of different leader election mechanisms for an op-stack chain. Leader election is desired as a feature because it can be used to make sequencing more decentralized. It does not guarantee that sequencing is more decentralized, so be careful when thinking about the coveted “decentralized sequencer”.

What is Decentralized Sequencing?

You could define a decentralized sequencer as 1) permissionless entry to proposer set, 2) some weighted probability of being elected leader based on a costly signal. Decentralized sequencing is a spectrum. Introducing PBS can slightly decentralize sequencing, but it will not give the property of censorship resistance because the single sequencer has the right to censor any block that they choose. Decentralized sequencing gives the property of censorship resistance, as if transactions are being censored then its possible for new sequencers to join the network and begin to include transactions that were previously being censored.

What is the MVP for leader election?

Allow the batch inbox to be a smart contract and check in the derivation pipeline that the transactions do not revert. This would allow developers to experiment with leader election mechanisms that are based on an L1 smart contract. This would require a few changes:

  • Gas estimation when creating the batch instead of computing the gas limit client side based on the intrinsic gas in the batcher
  • Batcher needs to be able to know when its the leader and when its not, so create a standard interface that all smart contracts used as the batch inbox must implement, so the batcher can submit batches only when it is its turn
  • Modifying the [derivation pipeline](https://github.com/ethereum-optimism/optimism/blob/edb6603feb5f49f2fdfe41f4b39c91695ed81ff8/op-node/rollup/derive/calldata_source.go#L107) to ensure that reverted transactions are not included, the receipts are necessary for this in the linked context
  • Remove the check that the tx.from is equal to the batch sender when the [batcherHash](https://github.com/ethereum-optimism/optimism/blob/develop/specs/system_config.md#batcherhash-bytes32) is version 1. The version is just the first byte, I don’t see a reason as to not just use bytes32(1) although there may be useful information that could be included in the bytes following the version byte
  • Pull the number of L2 blocks per channel into consensus and make it very easy to determine when channels open/close from the smart contract
  • A simple smart contract that implements a permissioned set of leaders and round robins between them as a proof of concept
  • Nice to have: add the batch inbox to the system config with a setter so it can be modified via governance and emit an event when it changes so the leader election mechanism can be dynamically changed by governance
  • Nice to have: a way for the fee recipient to be able to be set by the current sequencer, right now it is deterministic and always set to a predeploy

A contract that is a batch inbox should not read the full batch into memory, it should simply implement a fallback function and then do logic based on the msg.sender and be able to track channels. Once certain design patterns are proven out in the smart contract, they can be pulled into the derivation function similar to “precompiles” and then op-stack users can configure which sort of leader election mechanism they would like. Experimentation for leader election can now happen outside of the core protocol in parallel by anybody that wants to work on it.

How will this RFP help the Collective progress towards technical decentralization?

Leader election opens up the ability for multiple entities to act as a sequencer and produce canonical blocks. When there is only a single sequencer, there is risk that this sequencer can use its privilege to extort users through censorship or toxic MEV extraction. When there are multiple sequencers, the chain is as censorship resistant as the least censoring sequencer. Due to the complex second order effects of leader election, it is difficult to know for sure the ideal mechanism. This RFP is specifically designed to allow many possible leader election mechanisms to be expressed in smart contracts, so that teams can work on them in parallel outside of the core protocol. Then over time we can “enshrine” the mechanisms that work the best into the core protocol itself. The best mechanisms will be able to handle toxic MEV extraction and make it less of an issue for end users.

What is required to execute this RFP?

Minor changes to many parts of the codebase is required for this RFP to be implemented fully. Ideally, this code is behind a hardfork flag so that the code can still sync existing networks. See the “What is the MVP” section above for details

What milestones should be tracked to determine successful completion of this RFP? How should the Foundation measure progress towards this RFP?

  • Derivation pipeline changes as described in “What is the MVP”
  • Batcher changes as described above
  • Optimism system smart contract changes for the system config
  • Batch inbox smart contract that implements round robin
  • Running the local devnet with multiple batchers running, both using the same op-geth

The bullet points are the general chunks of work involved in the project and can act as milestones for development. Progress towards the completion of this RFP should be based on working code where working is defined as not production ready (not optimized, no metrics, not audited or thoroughly tested) but operating as expected based on enough test coverage to see it working and a final deliverable of a demo on the local devnet where it is obvious that multiple batchers are working in parallel.

How should the community measure impact upon completion of this RFP?

The community should measure impact of this RFP based on it removing one of the centralization bottlenecks from the system. The system is as centralized as the most centralized aspect. The lack of fault proofs is the most centralized aspect (a single party can rug the network) so we are working on that, but the lack of leader election will be the most centralized aspect of the system after fault proofs ship.

This also opens up the design space for people to experiment with leader election mechanisms, which will be very fun. We can imagine some experiments but guarantee people will come up with stuff we never imagined.

Application instructions

To apply for this RFP, please complete the form in the expandable section below and leave your response as a comment on this issue thread below. Submissions will be open until June 28, at which time the Foundation will review all submissions and select up to three individuals/teams to complete the work defined here.

Submission form

Copy the entire application below and leave a comment on this issue with your answers completed. A representative from the Optimism Foundation may reach out using the contact info provided to request more information as necessary.

Foundation Mission (RFP) Application

Please verify that you meet the qualifications for submitting at the above Tier

  • Alliance Lead: Please specify the best point of contact for your team
  • Contact info:
  • L2 recipient address:
  • Please list the members of your Alliance and link to any previous work:

Read more about Alliances here

What makes your Alliance best-suited to execute this Mission?

  • [...]
  • [...]

Please describe your proposed solution based on the above Solution Criteria (if applicable):

  • [...]
  • [...]

Please outline your step-by-step plan to execute this Mission, including expected deadlines to complete each peice of work:

  • [...]
  • [...]

Please define the critical milestone(s) that should be used to determine whether you’ve executed on this proposal:

  • [...]
  • [...]

Please list any additional support your team would require to execute this mission (financial, technical, etc.):

  • [...]
  • [...]

Grants are awarded in OP, locked for one year. Please let us know if access to upfront capital is a barrier to completing your Mission and you would like to be considered for a small upfront cash grant: (Note: there is no guarantee that approved Missions will receive up-front cash grants.)

  • [...]

Please check the following to make sure you understand the terms of the Optimism Foundation RFP program:

  • [ ] I understand my grant for completing this RFP will be locked for one year from the date of proposal acceptance.
  • [ ] I understand that I will be required to provide additional KYC information to the Optimism Foundation to receive this grant
  • [ ] I understand my locked grant may be clawed back for failure to execute on critical milestones, as outlined in the Operating Manual
  • [ ] I confirm that I have read and understand the grant policies
  • [ ] I understand that I will be expected to following the public grant reporting requirements outlined here

-- end of application --

bdresser avatar May 31 '23 18:05 bdresser

Hey, we have started work on this and would be interested in submitting soon. Is there a way to ask some question relating to this RFP on the Optimism discord?

jommi9 avatar Jun 16 '23 10:06 jommi9

hey @jommi9, if you head to the #Alliances channel in Discord, there's a thread for this RFP where you can ask questions. (Link here.) You're also welcome to ask on this issue.

cc @tynes

bdresser avatar Jun 16 '23 21:06 bdresser

Alliance Lead: @MeerKatDev Contact info: [email protected], [email protected] L2 recipient address: 0x6e5060047e69f0EF6881084355E606d9F3a5a421 Please list the members of your Alliance and link to any previous work:

  • Stoyan Kirov (Github, Linkedin) is a Software Engineer at Eiger. In his 10+ years of software development experience, he's worked in multiple industries ranging from revolutionising public transportation, to distributed cryptography, and fintech. Stoyan has a background in Go, JS and working with infrastructure, while in Eiger he's worked on a part of a blockchain project utilising confidential computing. He's currently mastering Rust and gaining a deeper knowledge of various blockchain projects.
  • Luca Campobasso (GitHub, LinkedIn) is a Software Engineer at Eiger. He has extensive experience with functional languages (7 years), especially with Scala and Elixir, mostly focused on backend systems design. He’s been working on research and development for several projects within the company helping in identifying new research opportunities and taking those ideas to full fledged implementations.

Eiger already has extensive experience developing large infrastructural projects. Some chosen examples:

  • The SnarkOS for Aleo (repo), the privacy-focused smart contract L1 platform going into production this summer.
  • OrbitDB (repo) is a distributed, p2p, serverless database.
  • The rust implementation of IPFS (repo), now archived.
  • The main implementation of Interledger in Rust (repo), which we developed and maintained.
  • Ziggurat (repo) - A Network protocol testing framework for ZCash, XRP, and Algorand. Notably, critical network vulnerabilities were found and reported to the core teams.

What makes your Alliance best-suited to execute this Mission? The promise of web3 is to upgrade the very foundations of our society – from money, finance, and governance to media, gaming, and science. To deliver on that promise, decentralised technologies are to be integrated into the everyday experiences of billions of people. For engineering, this is a mountain of a challenge.

Eiger was founded to develop infrastructure for web3 mass adoption. We help technology companies improve and integrate the core technologies of web3 to meet the climbing demands for scale and performance. We currently employ 30+ senior web3 engineers across the globe and work with some of the most ambitious organisations in the industry, including Forte, Aleo, and XRP Labs, to name a few. Eiger is part of Equilibrium Group, a blockchain powerhouse founded in 2018, composed of three entities:

  • Eiger: high-value add engineering services
  • Equilibrium Labs: applied research, proprietary products, and investing in early-stage ventures
  • Membrane Finance: infrastructure and services for EUROe, a euro-backed stablecoin

Please describe your proposed solution based on the above Solution Criteria (if applicable):

At the moment, the Op Stack works with a single sequencer and batcher. We will create a fork of the Optimism codebase with some minimal yet crucial changes, forming a proof of concept for version of the OP Stack that allows for block creation by multiple leaders, chosen at random or similar. The other objective of this proposal is to create a minimal PoC on which developers may experiment with different leader election mechanisms on L1.

This kind of leader selection will set a precedent for increasing the degree of decentralisation of an op-stack chain, and starting the work towards thinking about a “decentralised sequencer”.

Please outline your step-by-step plan to execute this Mission, including expected deadlines to complete each piece of work:

The development will last five weeks. Since we cannot assume a starting date, we’ll be giving the milestones in weekly objectives. The work will start within two weeks of approval. The plan is the following:

First week

  • Fork the codebase repository, on which we are going to work for the whole development timeline
  • Get to know the codebase and thoroughly examine the parts which are important for the project. Adjust the development plan based on findings.

Second week

  • At the moment, the Batch inbox is just a simple address on L1 in the eyes of the consensus nodes. We will change this around by creating a standard interface that modifies the Batch inbox to be a smart contract on L1 (BISM). This will also allow us to add programmable features such as notifying the single batcher to know when its turn arrived, among others.
  • Remove the check that the signer of the transaction tx.from (like here, and wherever else it’s needed) is equal to the batch sender when the batcherHash is version 1.

Third week

  • Add to the BISM gas estimation capabilities. The batcher needs to estimate gas when creating the batch, from inside the smart contract, instead of computing the gas limit client-side based on the intrinsic gas in the batcher.
  • Modify the derivation pipeline to ensure that reverted transactions are not included.

Fourth week

  • Write a smart contract (LSM) having a state storing a permissioned set of batchers (leaders), where the smart contract may randomly chooses one among them, and sets one of them as the batcher for the coming period. Create this as a template so that it’s easier for developers to change the picking heuristic.
  • Use the number of L2 blocks to determine when channels must be opened or closed by the BISM.

Fifth week

  • Finishing up the features above if something is still left to perfect, or solve any standing issues.
  • BONUS: add the batch inbox address to the system config with a setter so it can be modified via governance and emit an event when it changes so the leader election mechanism can be dynamically changed by governance.
  • BONUS: introduce in the LSM or BISM a method to set the fee recipient by the current leader.

Please define the critical milestone(s) that should be used to determine whether you’ve executed on this proposal:

  • The repository was forked
  • The development team formed a reasonable plan from beginning to end.
  • There is a standard interface for batchers, containing the needed functionalities for the remainder of the development
  • The check for the transaction origin as the batcher is removed when setting batcherHash to 1 (at least one regression test per each check found)
  • The gas calculation matches the new rules (regression test)
  • Check through a regression test that there are no reverted transactions in the derivation pipeline.
  • Appropriate tests for the included smart contracts
  • Documentation of the changes. On it, a user should learn:
    • how to form a batcher contract and deploy it
    • how to set the batcher as the batcher contract for the deployed node
    • what to expect from the changes, in more general terms

Please note that even though the milestones above describe implementations of working code, the aim is not yet to have the output be production ready - the final deliverable should be thought of as a demo working on a local devnet where multiple batchers work in parallel.

Please list any additional support your team would require to execute this mission (financial, technical, etc.):

At this moment we do not think we will need technical or financial support beyond the amount awarded by the grant.

Grants are awarded in OP, locked for one year. Please let us know if access to upfront capital is a barrier to completing your Mission and you would like to be considered for a small upfront cash grant: (Note: there is no guarantee that approved Missions will receive up-front cash grants.)

We agree with this policy, and there is no need for an upfront cash grant.

Please check the following to make sure you understand the terms of the Optimism Foundation RFP program:

  • [x] I understand my grant for completing this RFP will be locked for one year from the date of proposal acceptance.
  • [x] I understand that I will be required to provide additional KYC information to the Optimism Foundation to receive this grant
  • [x] I understand my locked grant may be clawed back for failure to execute on critical milestones, as outlined in the Operating Manual
  • [x] I confirm that I have read and understand the grant policies
  • [x] I understand that I will be expected to following the public grant reporting requirements outlined here

MeerKatDev avatar Jun 20 '23 16:06 MeerKatDev

  • Alliance Lead: Norswap
  • Contact info: telegram @norswap / [email protected]
  • L2 recipient address: 0x76188E6a9713433c98def583B8d616B081BEf958
  • Please list the members of your Alliance and link to any previous work:
    • Norswap
      • Ex Optimism core dev, reviewed the OP stack codebase in full through leaving Optimism last December, wrote most of the specs at that point in time as well, and also hacked on the codebase more recently.
      • 0xFable — fully on-chain trading card game, under development
      • RIPTIDE — NFT distribution mechanism via booster pack / lootboxes, ETHGlobal Lisbon prize winner
      • Duck Duck Destruction — battle bots game with MUDv2
      • Seqsy Sequencers — decentralized sequencer prototype, ETHTaipei second place winner
      • The Pondering Orb — capital efficient prediction market
    • Grapebaba
      • Built Hildr which is an alternative op-node implementation in Java 20 with GraalVM native, adding another option besides the offical op-node and Magi from a16z. Very knowledgeable on the inner working of the op-node and execution client.
      • Sui4j — Sui4j is a robust, reactive, type safe Java library for working with Smart Contracts on the @MystenLabs/sui network.
      • Hildr — An OP Stack rollup client written in Java 20 with graalvm native.
      • Battleship-move — A zk battleship game in Move on @MystenLabs/sui network.
    • Kai
      • Smart contract and ZK developer for the past two years, with a focus in the DeFi derivatives space and have also been actively building privacy infrastructure in DeFi. Previous life as a academic researcher in decentralized identity and as a backend engineer in web2 database scaling.
      • ZrcLib — Open source library for building private DeFi primitives on EVM compatible chains
      • Privacy-preserving Paymaster — ERC4337 paymaster module to privately withdraw from a shielded pool

What makes your Alliance best-suited to execute this Mission?

  • Pre-existing extensive experience with the OP stack (both Norswap & Kai Chen)
    • Norswap wrote a lot of the Bedrock specification and reviewed the code multiple times
    • Kai chen is writing an alternative op-node client implementation in Java, called Hildr

Please describe your proposed solution based on the above Solution Criteria (if applicable):

The above description is pretty extensive, we don't have much to add to it.

We would like to figure out a clean interface on the node side to complement the validity conditions in the batch smart contract. This would make it easy to migrate between smart-contract based validity conditions (more costly) in gas to node-side validity conditions, also accomodating a mix of both to enable experimentation and accomodate a variety of use cases.

Please outline your step-by-step plan to execute this Mission, including expected deadlines to complete each piece of work:

  1. Come up with a tentative specification for the work — 31 August 2023
    • sequencer registry smart contract interface
    • specify the changes necessary to have fixed-size channels such that every channel is the responsability of a single leader
    • specify how the batcher and the op-node will query the sequencer registry to know if it's their turn to put together and submit a block
      • possibly this mechanism can also be used to validate the batch proposer instead of contract-side validation
      • as well as to get the fee recipient specified by the leader on L1
  2. Write the simplest possible MVP, as a way to elucidate problems that may have been overlooked. — 30 September 2023
    • e.g., only two sequencers, alternating round-robin, using single-block channels, ...
  3. Expand to the full scope — 31 October 2023
  4. Take care of the all the unforeseen stuff 🙃 / built-in slack — 30 November 2023

Our baseline estimation for the amount of work required is 3 month of engineering work, split between the three of us. We will be working on this in parallel with other endeavours, but will prioritize this work as a matter of commitment.

Please define the critical milestone(s) that should be used to determine whether you’ve executed on this proposal:

Copied from the RFP:

  • Derivation pipeline changes as described in “What is the MVP”
  • Batcher changes as described above
  • Optimism system smart contract changes for the system config
  • Batch inbox smart contract that implements round robin
  • Running the local devnet with multiple batchers running, both using the same op-geth

Please list any additional support your team would require to execute this mission (financial, technical, etc.):

Access to people able to answer questions, including on most preferred direction when there is a choice, would be great.

Grants are awarded in OP, locked for one year. Please let us know if access to upfront capital is a barrier to completing your Mission and you would like to be considered for a small upfront cash grant: (Note: there is no guarantee that approved Missions will receive up-front cash grants.)

It is not an obstacle, though we would like to be considered for upfront cash grant, as we currently do not have revenues.

Please check the following to make sure you understand the terms of the Optimism Foundation RFP program:

  • [x] I understand my grant for completing this RFP will be locked for one year from the date of proposal acceptance.
  • [x] I understand that I will be required to provide additional KYC information to the Optimism Foundation to receive this grant
  • [x] I understand my locked grant may be clawed back for failure to execute on critical milestones, as outlined in the Operating Manual
  • [x] I confirm that I have read and understand the grant policies
  • [x] I understand that I will be expected to following the public grant reporting requirements outlined here

norswap avatar Jun 21 '23 16:06 norswap

Alliance Lead: Ben Fisch Contact info: [email protected] L2 recipient address: 0x07d587497477F4ED326C02810670Ad45e8A9916F

Please list the members of your Alliance and link to any previous work: Espresso Systems Technical Leads:

  • Ben Fisch (LinkedIn): Espresso Systems CEO and cofounder. Assistant professor of computer science at Yale University. Previously completed a PhD in computer science at Stanford University working with Dan Boneh, where his research focussed on cryptographic proof systems and blockchain scalability. His seminal work on Verifiable Delay Functions (VDFs) and Proofs-of-Replication (PoReps) in 2018 were influential in the design of Filecoin, Chia, and Ethereum’s beacon chain.

  • Benedikt Bünz (GitHub, LinkedIn): Espresso Systems Chief-Scientist and cofounder. Incoming assistant professor of computer science at New York University (NYU). Previously a PhD candidate at Stanford University working with Dan Boneh. His research on Verifiable Delay Functions (VDFs) sparked the VDF Alliance, while Bulletproofs secure tens of thousands of transactions on Monero and Mobilecoin. His research on HyperPlonk is being integrated into the popular Halo2 library and frontend maintained by the Ethereum Foundation.

  • Fernando Krell (LinkedIn): Fernando is VP of Engineering at Espresso Systems. Before joining Espresso, Fernando obtained his PhD in computer science from Columbia University, where he specialized in cryptographic protocols with applications to private search mechanisms.

  • Nathan Yospe (GitHub, LinkedIn): Nathan is an engineer at Espresso Systems, fluent in rust and C++. Before joining Espresso, Nathan was the architect responsible for SAP's Netweaver MDM platform, and the first version of the HANA Vora distributed execution engine. Nathan implemented a full end-to-end unicode library in C++ and authored one of the first NLP ingestion systems (language training neural network), a distant precursor to modern LLMs like GPT-4. Nathan holds a Bachelor’s degree in Physics and Mathematics from the University of Hawaiʻi at Mānoa.

  • Mathis Antony (GitHub, LinkedIn): Mathis is a smart contract and systems developer at Espresso Systems, currently leading the development of the sequencer application. Before joining Espresso, Mathis founded CognAI, a machine learning consultancy to help customers leverage computer vision. In 2018 he joined the Blockchain consultancy Enuma Technologies to build a trustless Layer 2 Exchange. Mathis holds a Bachelor's degree in Physics from the École Polytechnique Fédérale de Lausanne (EPFL), and a Master's degree in the same field from the Hong Kong University of Science and Technology (HKUST).

  • Noah Golub (GitHub, LinkedIn): Noah Golub is a software engineer with experience in both blockchain and B2C. He has previously worked at Facebook, Vest (blockchain staking infrastructure), and Real Mental Health.

  • Jeb Bearer (GitHub, LinkedIn): Jeb Bearer is a software engineer. He has 5 years of experience building secure and reliable systems, focusing in C, Haskell, and Rust. Before joining Espresso, Jeb performed formal methods research and contributed to a real-time operating system for mission-critical embedded systems.

Espresso Systems Overview: Espresso Systems has extensive experience shipping fundamental infrastructure for web3 and blockchain systems. We typically use the MIT license, a highly permissive free software license, to accelerate ecosystem growth. A few examples include:

  • HotShot - a consensus protocol for transaction ordering, featuring high throughput and fast finality, while scaling to thousands of validators.
  • Espresso Sequencer - a decentralized sequencing and data availability layer for Ethereum rollups, which enables rollups to provide users with pre-confirmations prior to L1 settlement. At its core is a proof-of-stake deployment of HotShot and a HotShot light-client implemented in a smart-contract, which exposes an interface for other rollup smart-contracts to read from.
  • HyperPlonk - a zero-knowledge proof system designed for large circuits as it removes requirements for Fast Fourier Transforms (FFTs). We’re collaborating with the Ethereum Foundation to integrate HyperPlonk into the popular Halo2 library and frontend.
  • Jellyfish - a toolkit of cryptographic primitives implemented in Rust, ranging from hash functions to accumulators to zero-knowledge proof systems. Includes PLONK, a zk-SNARK with universal setup, which is feature complete and fast.
  • Configurable Asset Privacy for Ethereum (CAPE) - a smart contract application that enables Ethereum based asset creators to configure who can view information on-chain, e.g. sender and receiver address, asset amount, and type of asset across several actions (send, receive, custody).
  • VERI-ZEXE - decentralized private computation that is programmable and verifiable with 9x speed up for transaction generation and 2.6x saving on memory usage compared to ZEXE.Read more about Alliances here

What makes your Alliance best-suited to execute this Mission? Through our work at Espresso Systems, we are dedicated to the development of the rollup ecosystem. Rollups promise to bring higher throughput and lower fees to Ethereum. Most rollups have ambitious roadmaps that include new codebases, features and plans to decentralize. Espresso Systems is building a decentralized shared sequencing network as a public good that allows rollups to decentralize without compromising on performance and speed.

Our deep experience researching and developing sequencer solutions for the rollup space makes our team uniquely well-suited to undertake this Mission and contribute to OP and the Superchain future.

Please describe your proposed solution based on the above Solution Criteria (if applicable): We suggest making as flexible and modular an interface as possible for experimentation in election models. We plan to implement this, either in a disclosed fork of ethereum-optimism/optimism or against a designated branch. We intend to deliver an update that fully incorporates the set of points outlined in the “What is the MVP for leader election?” section of the RFP, including the elements noted as “nice to have”. At a high level, this means: the Solidity interface, a PoC contract, updates to the batch header format, changes to batcher submission, changes to the derivation pipeline, and corresponding updates to documentation and testing. We will start with those precise elements, but if the timeframe can accommodate, we hope to include additional features. In particular, we want to write a second contract with a dissimilar selection strategy as a forcing function for generality of the interface.

Ultimately, we envision this as the first step of a longer term collaboration. We acknowledge the precise scope of the RFP and intend to provide an interface that supports the requested leader selection mechanism, and is also fully capable of supporting a broad range of other mechanisms for sequencer decentralization. This includes alternatives to round-robin leader election, such as solutions utilizing VRFs and/or VDF-based randomness beacons, as well as consensus protocols to be run among all sequencer nodes. One advantage of a consensus protocol over a simple leader selection is the ability to provide pre-confirmations (aka soft-confirmations) before Ethereum finalizes the block, which centralized sequencers are able to provide today.

Please outline your step-by-step plan to execute this Mission, including expected deadlines to complete each piece of work:

In the following paragraphs we describe the tasks required to meet the MVP requirements. For each task, we provide time estimates for completion, corresponding to one engineer as a primary developer plus an additional reviewer. If performed sequentially these tasks would add up to 11 weeks of development, but we note that T2, T3 and T4 can be done in parallel resulting in 7 weeks of development in total.

T1: We will define an interface, in Solidity, for a batch inbox that supports leader election strategies. This interface will be reasonably thought out, but could be updated before the final delivery, based on discoveries during implementation of the remainder of this proposal. We will include calls for checking if an address or this instance is eligible to submit batches during the current round, and possibly for checking whether the same address has a known span in which it will be, ending at or after the current round, and returning that span if it does. Estimate: [one week].

T2: Next, we will implement a reference contract implementing this interface, which will use a fixed list of predefined batcher addresses (specified in deployment) to select the current eligible batcher as a round robin function, as requested in the RFP. The size of the list should be specifiable at the time of deployment. We will write this with an expectation that functions (and things like tweaks to batch headers) that are not yet available can be mocked and hooked up once the remaining tasks are complete. Estimate: [two weeks].

T3: We will then update the batch submitter to target an instance of this new interface. Initially, the batch inbox address may not be dynamically configurable via governance, but since this is a desired feature we will either include it or make its inclusion minimally disruptive. This task includes many prerequisite subtasks, such as updates to the batch header format, replacing the call to IntrinsicGas, updating the BatcherHash, removing the built-in batcher address check, and making a pre-submission check to confirm that this instance is currently permitted to submit. Estimate: [two weeks].

T4: We will update the derivation pipeline, per the MVP. For this version, we will not attempt to make any significant changes to support pipelining between batchers. Estimate: [four weeks].

T5: We will prepare and execute a series of test scenarios around a set of batchers, including runs with two alternating batchers and three round robin batchers, and tests that stress the boundary conditions (the last point a batcher is eligible/first point it is ineligible, etc) around batch submission timing. This will also include failure scenarios (an invalid batcher address, channel never closes during window, etc.). Estimate: [two weeks].

Nice-to-have features Upon completion of the MVP task above, we will turn our attention to additional features, including the nice-to-have points in the RFP and additional items proposed by Espresso Systems. In particular, if time allows we would work on the following features:

  • T6: Batch inbox address in system config. Estimate: [one week].
  • T7: Specification of a fee address set by sequencer. Estimate: [one week].
  • T8: We will add a hardfork flag around the deltas in our branch, and prepare this for submitting back to the OP trunk. Estimate: [immediate, with support].
  • T9: An alternative contract using a contrived, but plausible, selection function implementing the same interface. This new contract implementation aims at evaluating the generality and usefulness of the proposed interface. Estimate: [two weeks].
  • T10: We will confirm that documentation reflecting the changes we have made is up to date. Estimate: [one week].

Follow-up project The Espresso team envisions a follow up project that aims to decouple ordering in the L1 contract from available data. In particular, the contract only needs to receive unique references to data blobs (e.g., hashes) that are certified available (whether by Ethereum itself via EIP-4844 transactions or alternative DA layers).

Please define the critical milestone(s) that should be used to determine whether you’ve executed on this proposal:

  • The Batch Inbox interface and at least one reference contract exists
  • The changes needed to support this have been added to the batcher
  • The changes needed to support this have been added to the derivation pipeline
  • Representative and comprehensive tests have been written and can be successfully executed
  • A demonstration of this in operation with multiple batchers can be shown
  • The implementation is ready for review and integration

Please list any additional support your team would require to execute this mission (financial, technical, etc.): Espresso Systems is new to working with the OP codebase (though ramping up!). We’d like to request occasional technical support, e.g. Q&A, identifying issues, testing and bug bashing.

Grants are awarded in OP, locked for one year. Please let us know if access to upfront capital is a barrier to completing your Mission and you would like to be considered for a small upfront cash grant: (Note: there is no guarantee that approved Missions will receive up-front cash grants.)

We agree to this provision and no upfront cash grant is required.

Please check the following to make sure you understand the terms of the Optimism Foundation RFP program:

  • [x] I understand my grant for completing this RFP will be locked for one year from the date of proposal acceptance.
  • [x] I understand that I will be required to provide additional KYC information to the Optimism Foundation to receive this grant
  • [x] I understand my locked grant may be clawed back for failure to execute on critical milestones, as outlined in the Operating Manual
  • [x] I confirm that I have read and understand the grant policies
  • [x] I understand that I will be expected to following the public grant reporting requirements outlined here -- end of application --

ben-a-fisch avatar Jun 28 '23 05:06 ben-a-fisch

Hi all, this is not a direct proposal to implement what is described in the above ask, however we have spent considerable time researching the RFP and speaking with invested parties, and would like to make a submission in support of @norswap and team’s proposal. Obol is keen to be involved in the decentralisation of sequencers for OP-stack, and would like to provide feedback on any selected bid's implementation of a leader election framework, with the hope that what we would like to prototype on top of the effort is achievable with the selected design.

  • Alliance Lead: @xenowits
  • Contact info: telegram @xenowits, email: [email protected]
  • L2 recipient address: 0x43F641fA70e09f0326ac66b4Ef0C416EaEcBC6f5

Please list the members of your Alliance and link to any previous work:

  • This is a proposal from the Obol Labs core team, a team of 26 that has been building in the space for 2+ years.
  • Some of our developers that will be most directly supporting this effort are: @xenowits, @0xdhruv and @samparsky
  • @xenowits
    • Protocol Engineer at Obol
    • Core contributor to Obol’s DVT client, charon
  • @dB2510
    • Protocol Engineer at Obol
    • Core contributor to Obol’s DVT client, charon
  • @samparsky
    • Smart contracts lead at Obol
    • Ex-Advanced Blockchain AG. Solidity lead.
  • @oisinkyne
    • CTO and Co-Founder at Obol.
    • Built blockdaemon's Eth2 staking solution

What makes your Alliance best-suited to execute this Mission?

  • As we outlined above, we came to the conclusion that we do not have the same familiarity with the OP-stack codebases as other responders to this RFP, among which include previous core-devs, however we do have a strong background in building distributed systems for Ethereum, and would like to provide input on the types of leader election mechanisms that might be built in future on top of this effort.
  • Obol Labs has developed the idea of Distributed Validator Technology, allowing for L1 Ethereum validators to be run by multiple independent entities co-operatively. The core design goal Obol would like to see achieved at L2; is that Sequencers are not just monoliths run by single entities, but instead, their design is MPC-compatible, such that a single logical sequencer can be run by independent parties. We believe making groups of entities run sequencers together reduces liveness, safety, and censorship risks.
  • We recently helped publish a research piece on how Distributed Sequencer Technology (DST) may be applied to the OP stack, see figment capital’s post on the matter.

Please describe your proposed solution based on the above Solution Criteria (if applicable):

Our proposal for how OP chains should approach leader selection for their sequencer set is by developing an Ethereum L1 equivalent design as much as is feasible, and then experimenting with improvements from there. The design idea could be described as ‘an OP sequencer node should be as easy to run as an L1 validator node except with higher specs’.

This intention is aligned with the Optimism bedrock design of only communicating with EL’s through the engine API, enabling OP chains to support many execution clients. We believe that OP sequencing nodes should have their signing interface abstracted out of the core binary, and that signer should communicate with OP-node as if it were speaking the ValidatorRequiredAPI, but with simplifications. (such as removing attestation duties and only doing proposals in a round robin or L1 randomness based approach)

We think a minimal prototype could be achieved as part of the MVP described here of round-robin based leader election.

We think L1 alignment is a worthwhile strategy to undertake as it would allow the OP ecosystem to tap into a wealth of research that has been developed for the Ethereum PoS system, such as:

Please outline your step-by-step plan to execute this Mission, including expected deadlines to complete each piece of work:

  • The majority of the effort will be undertaken by another team selected to bring the OP codebases to a state where design ideas like ours can be prototyped. The Obol core team will make themselves and their advisors available to the winning team to provide design feedback.

Please define the critical milestone(s) that should be used to determine whether you’ve executed on this proposal:

  • Not applicable

Please list any additional support your team would require to execute this mission (financial, technical, etc.):

  • The Obol team would like to offer support to the winning bid rather than requiring support.
  • We have extensive experience in golang and in solidity which the teams can leverage for design and code review

Grants are awarded in OP, locked for one year. Please let us know if access to upfront capital is a barrier to completing your Mission and you would like to be considered for a small upfront cash grant: (Note: there is no guarantee that approved Missions will receive up-front cash grants.)

  • Upfront capital is not a blocker for Obol participating in this effort, nor are we seeking a grant. If any grants were to be allocated to Obol we would ask they be retrospective based on contribution made rather than being up-front at the point of commitment.

Please check the following to make sure you understand the terms of the Optimism Foundation RFP program:

✅ I understand my grant for completing this RFP will be locked for one year from the date of proposal acceptance.

✅ I understand that I will be required to provide additional KYC information to the Optimism Foundation to receive this grant

✅ I understand my locked grant may be clawed back for failure to execute on critical milestones, as outlined in the Operating Manual

✅ I confirm that I have read and understand the grant policies

✅ I understand that I will be expected to following the public grant reporting requirements outlined here

OisinKyne avatar Jun 28 '23 15:06 OisinKyne

Submissions for this RFP are now closed. Thanks to everyone who submitted a proposal!

Someone from the Optimism Foundation will reach out on or shortly after July 13 to communicate which proposal(s) have been accepted and schedule a kickoff. We may also reach out individually using the contact info you've provided if we want to discuss your proposal in more detail.

In the meantime, feel free to tag me here or DM ([email protected]) with any questions.

bdresser avatar Jun 28 '23 19:06 bdresser

Hi all – we are extending the selection deadline to July 20 to allow time for review. The Optimism team may be in touch to discuss the proposals above. Again, feel free to reach out with questions.

bdresser avatar Jul 10 '23 18:07 bdresser

Hi folks – thank you again for the excellent proposals here.

We're excited to share that we will be moving forward with the proposal from @ben-a-fisch and Espresso Systems. Their thoughtful submission combined with their track record in this area make them an excellent fit for the project.

@ben-a-fisch we will be in touch directly to discuss the project and expectation in more detail.

For all other teams here: we'd love to help you find the right way to contribute to the Optimism Collective. See the rest of the issues in the Ecosystem Contributions repo for ideas, suggestions, and other possible projects. And stay tuned for more RFPs posted in the next few months. Thank you again for your proposals.

bdresser avatar Jul 18 '23 15:07 bdresser

We are pleased to update the community on the latest developments. Last week marked the starting point of the tasks outlined in our proposal. Our implementation is publicly accessible in an Optimism fork, which can be found at https://github.com/EspressoSystems/op-leader-election. For project management, we have divided the proposal milestones into public GitHub issues, which we plan to address in the upcoming weeks. The current status on these tasks is listed below:

T1: Leader Election Interface (Done) PR T2: Reference Smart Contract Implementing interface (Next)

T3: Update Batch submitter (In progress)

T3.1: Update Batch Header (Done) PR T3.2: Replace Call to IntrinsicGas (In progress) T3.3: Update BatcherHash (Not started) T3.4: Presubmission check (Not started) T3.5: Removing built-in batcher (Not started) T3.6: Build/Update test for batch submitter (Not started) T4: Update Derivation Pipeline (In Progress)

T4.1: Identify reverted transaction (Done) T4.2: Pull receipt, scan transaction & filter them (Next) T5: Test Scenarios (Not started)

Stretch Goals (Not started)

T6: Batch inbox address in system config T7: Spec of fee address set by sequencer T8: Hardfork flag T9: Contrived contract implementing iface T10: Documentation check

ben-a-fisch avatar Aug 07 '23 19:08 ben-a-fisch

Dear community, this is Philippe from the Espresso Systems team. During the last two weeks we have been working on the following tasks:

  • T2: Reference Smart Contract #3 The reference smart contract for leader election is implemented. While working on it, we discussed future improvements to the leader election API that are documented in T9.1 Improvements to API #37.
  • T4 Update derivation pipeline #11: the derivation pipeline logic has been updated to take into account parsing / filtering of the batch inbox contract transactions.
  • T3: Update Batch Submitter #4 We are actively working on adjusting the code of the batch submitter to interface with the new batch inbox contract and support multiple batchers. This macro task will be our main focus during the next week.
  • T5.1 Design the integration tests #32 we started to specify the integration tests needed to validate the behavior of the whole system in presence of multiple leaders.

Stay tuned!

philippecamacho avatar Aug 18 '23 20:08 philippecamacho

Dear community, it is a pleasure to share some updates with you regarding our progress. The great news is that all tasks from T1 to T4 are done, with the exception of T3.4 which is under review and should be closed early next week. Now we will start working on the integration tests and aim for completing them by mid september according to the original plan. Here are more details:

Enjoy the week-end!

philippecamacho avatar Sep 01 '23 19:09 philippecamacho

Dear community, time for an update!

T3.4 turned out to be more complex than expected. The reason is that in a setting with multiple leaders, it becomes more difficult to time correctly the opening and closing of a channel. In particular with the initial implementation of the Leader Election contract, batchers may likely not be able to open and close a channel during the propagation of a single L1 block. To overcome this problem we adjusted the contract in order to assign several consecutive L1 blocks to each leader. With this change and the implementation of the new timing logic, we were finally able to close the last feature task of the plan.

On the testing side, we have extended the infrastructure in order to support multiple batchers and initialize the Leader Election contract with corresponding funded addresses. So, despite the delay w.r.t. to the original plan, we are finally in condition to implement the integration tests.

Thank you!

philippecamacho avatar Sep 15 '23 18:09 philippecamacho

How will this project compare to conduit elector tech who are also already one of the main deployers of OP Rollups? https://twitter.com/KAndrewHuang/status/1706685729274007819

jommi9 avatar Sep 27 '23 11:09 jommi9

Hi @jommi9 - The work done related to this RFP represents the first steps to being able to use alternative approaches to sequencing within the OP Stack, building the end points that other sequencer approaches can plug into.

From our understanding of their announcement, Conduit has developed an approach to sequencing that guarantees better uptime than some other options out there.

jillrcarlson avatar Sep 27 '23 16:09 jillrcarlson

Dear community,

During the last two weeks we have been focusing on the integration tests. We have implemented the first two ones (T5.2.1 and T5.2.2) and fixed a number of bugs in the process. Now we are tackling the other tests, in particular one of the most challenging, T5.2.8, which consists of validating that we can smoothly transition from a single leader regime to a multiple leaders regime. In parallel we have been thinking through the impact of multiple leaders on the logic for opening and closing channels and discussed our ideas with the OP Labs team. Next week we will continue working on the tests.

Enjoy the week-end!

philippecamacho avatar Sep 29 '23 21:09 philippecamacho

Dear Community,

It is our pleasure to announce that we have completed all the core goals of the RFP. We are very grateful to the OPLabs team for this opportunity to contribute to this major project in the space. Last Friday, we started discussing the next steps of this work, one possible direction being to make the OP stack easy to integrate with any kind of sequencer/DA layer.

We look forward to continuing to contribute to the OP ecosystem!

philippecamacho avatar Oct 17 '23 15:10 philippecamacho

@philippecamacho Hello, I have some questions based on your great work

  1. I haven't really read the op-stack codes but would the changes require some modifications with op-geth as well?

  2. Batch submission is moved from tx input data to contract input which would increase gas costs, could you tell the ratio for gas cost increase?

  3. Since using the contract to input batches, I think it is possible to modify the contract to emit events on every batch submission and listen for contract events instead of scanning every L1 block for L2 retrieval. See the issue also https://github.com/ethereum-optimism/optimism/issues/7933.

ghost avatar Nov 03 '23 13:11 ghost

Hey @kaliubuntu0206, thank you for your questions. Before sharing the answers, I want to highlight that this work has been done by several members of the Espresso Systems team and we also received great feedback and guidance from OP Labs!

  1. no.
  2. L1 dependent (space/execution ratio is not identical across testnets) but for a simpler contract on Ethereum, it looked like around 40% higher cost.
  3. Discussed this idea with people from OP at one point, and on their advice, deferred it for later.

Have a great week-end!

philippecamacho avatar Nov 03 '23 15:11 philippecamacho

Happy to keep discussing @kaliubuntu0206, would you like to join our Espresso Systems discord?

philippecamacho avatar Nov 03 '23 16:11 philippecamacho

Hey @kaliubuntu0206 - if you post in the #espresso-sequencer channel in Discord one of us on the team will be happy to chat there! Including Philippe if he is available :) See in you the chat!

jillrcarlson avatar Nov 03 '23 16:11 jillrcarlson

Hi @ben-a-fisch

My name is Julian and I am with the foundation. I am trying to get in contact with your team regarding your grants KYC/KYB. Would you please let me know an appropriate email to get in touch?

cc @jillrcarlson @philippecamacho

opjulian avatar Jun 06 '24 00:06 opjulian