[EPIC] Sequencing API

[Manav-Aggarwal]

• [ ] https://github.com/rollkit/rollkit/issues/829
• [x] #1724
• [x] Refactor sequencer like an endpoint so that it can:
  • Accept a Tx
  • create a batch post to DA
  • pull/subscribe for batch
• [ ] Refactor executor to:
  • Subscribe/pull batch
  • Talk to verifier for validating the batch
  • pull block time from sequencer header
  • Execute the batch to produce a header
• [ ] Implement batch verifier
  • Runs Shared Sequencer Light node
  • Runs DA light node
  • Verifies:
    • Batch is part of canonical SS chain
    • Batch is published to DA

[Manav-Aggarwal]

Needs from Astria:

• API links on:
  • pulling batch from the SS
  • code pointers from the conductor on where this is happening
• Getting Merkle proofs from the SS Namespace
• Verification of Merkle proofs
• Serialization/Deserialization and decompression of batch data
• Composer submission to SS
• pull block time from sequencer header

@joroshiba

[joroshiba]

For grabbing data from the sequencer: we have a grpc service withGetFilteredSequencerBlock which returns the transactions and proofs required to validate data for a single rollup. Protobufs are posted to buf, directly link to rpc definition here

Data posting to celestia is generally batched, serialized as protobuf data and compressed via brotli with a compression setting of 5. There are, from the view of an individual rollup, two batches: Sequencer Metadata and Rollup Data

This documentation needs some updating with new data shapes, but the gist is correct: https://github.com/astriaorg/astria/blob/main/specs/data-flow-and-verification.md

Transactions can be submitted to composer (essentially a gas station) via grpc: https://buf.build/astria/composer-apis/docs/main:astria.composer.v1alpha1. The "rollup id" here is also used by conductor for knowing which rollup to read from sequencer and which namespace to read on celestia.

[gupadhyaya]

i think rollkit needs to use more generic sequencer interface, something like:

// SequencerInput ...
service SequencerInput {
    // SubmitRollupTransaction ...
    rpc SubmitRollupTransaction(SubmitRollupTransactionRequest) returns (SubmitRollupTransactionResponse) {}
}

// SubmitRollupTransactionRequest ...
message SubmitRollupTransactionRequest {
    // the unhashed rollup id
	bytes rollup_id = 1;
	// the raw data bytes of the rollup transaction
	bytes data = 2;
}

// SubmitRollupTransactionResponse ...
message SubmitRollupTransactionResponse {
}

// SequencerOutput ...
service SequencerOutput {
    // SubmitRollupTransaction ...
    rpc GetNextBatch(BatchRequest) returns (BatchResponse) {}
}

// BatchRequest provides the last batch while requesting for the next batch
message BatchRequest {
    repeated bytes transactions = 1;
}

// BatchResponse contains the transaction batch that is last sequenced
message BatchResponse {
    repeated bytes transactions = 1;
}

// BatchVerifier
service BatchVerifier {
    // VerifyBatch ...
    rpc VerifyBatch(BatchRequest) returns VerificationResponse {}
}

// VerificationResponse
message VerificationResponse {
    bool success = 1;
}

and rollkit will use grpc client over the above interface for sequencing purpose. the individual sequencer implementations (like our centralized sequencer, astria seq, etc) can implement a grpc server serving the above requests. for instance, in case of astria,

• the composer implements the SequencerInput
• conductor implements the SequencerOutput & BatchVerifier

the reason behind making this more generic is obviously to support different kinds of sequencers and rollkit does not need to dwell into specific details about how the sequencer is implemented. we can create two repos centralized-sequencer and astria-sequencer in rollkit that connects this interface and respective implementations to test and refine this generic interface. lmk what you guys think? @tzdybal @Manav-Aggarwal @joroshiba

[gupadhyaya]

sequencing api: https://github.com/rollkit/go-sequencing centralized-sequencer implementing the above interface: https://github.com/rollkit/centralized-sequencer

[joroshiba]

Quick notes here that I don't think the previous batch is a great way to index on what the next batch will be, should be some unique identifier. Potentially a number.

I'm also not sure what the point of the verifier is, why not just only respond with verified data? Is the idea to get the batch ASAP and then verify from DA?

There is also data missing here that I would think you would want from a batch, particularly the timestamp at which it was created. Otherwise you cannot have deterministic block derivation the timestamp would be generated later, requiring either another round of consensus or a centralized entity who produces the block everyone else executes. This kinda muddles the point of having a third party sequencer.

[gupadhyaya]

thanks @joroshiba for your response.

1. yes we don't need to use the batch itself, we could use merkle tree hash. the intention here is to not rely on any particular sequencer specific information (such as sequencer height). Let us know if hash works for you guys.

2. yes, we want to receive a soft-commitment based batch asap and calling verify later to check if 1) the batch is part of the sequencer canonical chain (using sequencer light client) and 2) the batch is in DA.

3. agree, we can use the sequencer timestamp. will update the api to return batch along with timestamp.

The whole idea here is to create this middleware consisting of composer and conductor (that only verifies, not create block or execute transactions) that are sequencer specific. This way rollkit does not need to be bothered with sequencer details.

[gupadhyaya]

Points after discussion:

• Currently only header producer is reaping mempool txs and submitting to centralized sequencer. Nodes can configure themselves to either 1) directly submit the tx to centralized seq or 2) gossip such that someone else will submit to centralized seq.
• Currently all nodes (header producer and full nodes) connect to centralized sequencer for submitting tx, getting next batch, and verifying batch. However, the node that is submitting tx only needs to connect to centralized seq, other nodes can use DA to get the next batch and verify the batch.

[yarikbratashchuk]

Points on based sequencing implementation (after discussion):

• SubmitRollupTransaction maps to go-da Submit call.
• GetNextBatch method calls go-da: GetIds and Get methods to get the the list of blobs which maps to the sequencing batch
• VerifyBatch returns true as we get batch from da during GetNextBatch.

[Manav-Aggarwal]

Based Implementation:

Mapping go-sequencing to go-da

1. SubmitRollupTx (RollupId, Tx): (error) -> Submit([]blob, gasPrice, namespace ): []Id

RollupId can be discarded. Tx is converted to a blob. gasPrice and namespace are configured in the implementation.

2. GetNextBatch(lastBatchHash, maxBytes): (*Batch, timestamp, error) -> GetIds(height, namespace): []Ids and Get(Ids[], namespace): []blobs lastBatchHash and a mapping to the corresponding height is stored on the implementation. Initially, this height is set to DAStartHeight. First, GetIds is called using the height after the lastBatchHash and namespace which is configured. Then, the return value of this: []Ids is passed to the Get method along with the namespace which is configured. Then, with the return value of this: []blob, we only reap the blobs that fit into maxByes, keep track of the pending blobs, and we convert these blobs to *Batch. The timestamp is something we need to add in the go-da interface. Any pending blobs should be prepended to the next call to GetNextBatch

3. VerifyBatch (batchHash): (bool, error) -> true This should always return true as we get batch from DA already during GetNextBatch so there's no need to verify proofs.

Configuration can look something like this:

config {
    gasPrice,
    namespace,
    DAStartHeight,
    RollupBlockTime,
    DaBlockTime
}

[gupadhyaya]

pending sequencing tasks

• [ ] https://github.com/rollkit/rollkit/issues/1842
• [ ] https://github.com/rollkit/rollkit/issues/1840
• [ ] https://github.com/rollkit/rollkit/issues/1828
• [ ] https://github.com/rollkit/rollkit/issues/1831
• [ ] https://github.com/rollkit/rollkit/issues/1832
• [ ] https://github.com/rollkit/rollkit/issues/1833
• [ ] https://github.com/rollkit/rollkit/issues/1834
• [ ] https://github.com/rollkit/rollkit/issues/1826
• [ ] https://github.com/rollkit/rollkit/issues/1819
• [ ] https://github.com/rollkit/rollkit/issues/1812
• [ ] https://github.com/rollkit/rollkit/issues/1811
• [ ] https://github.com/rollkit/rollkit/issues/1810
• [ ] https://github.com/rollkit/rollkit/issues/1809
• [ ] https://github.com/rollkit/rollkit/issues/1843