Draft HardForks.md and/or ProtocolVersioning.md

[nfrisby]

A document discussing how the node (mostly Ledger and Consensus) handle hard forks.

This is part of my going effort to justify simplifying the HFC and might also be useful for deriving some of the Cardano Blueprint.

[nfrisby]

Rendered

• (deprecated) https://github.com/IntersectMBO/ouroboros-consensus/blob/nfrisby/doc-triggers/docs/website/contents/for-developers/HardForks.md

• (current) https://github.com/IntersectMBO/ouroboros-consensus/blob/nfrisby/doc-triggers/docs/website/contents/for-developers/ProtocolVersioning.md

[nfrisby]

I just pushed up a rewrite, added for now as a second file.

There's probably some fat to trim still and maybe another dimension of the design to better emphasize, but I think it has a useful organization and flow. ProtocolVersioning.md

[jasagredo]

In general the document looks good, and I think there explanation is quite clear and I appreciate the fact that no code is actually depicted here.

[nfrisby]

• As of writing this latest draft, I started seriously considering changing the HFC envelope to contain the protocol version instead of the era tag.
• I eventually realized that the cardano-cli $era build command (et al) would pose a challenge: it would have to choose a protocol version to include in the new tx, but the user only specified an era. For today's eras, it'd be sound to use the lowest protocol version of that era, but --- with protocol versions in the CBOR --- it would be possible for intra-era hard forks to affect the codecs, and so this "use era's lowest protocol version" heuristic would no longer obviously be sound (we could choose to continue to not allow intra-era hard forks to alter codecs).
• After some study and thought for a few days, I think the answer is: TxSubmission and LocalTxSubmission should be exchanging transactions that don't have any tags. The node's TxSub and/or Mempool logic would introduce the tag based on its current ledger state when the untagged tx was received --- which also determines which codec the node would use to (attempt to) parse the incoming transaction.
• The only corner case I can see where this would degrade behavior is (I'm describing this with era tags, but it's the same with protocol version tags): - The node's selection advances from era X to era Y. - An untagged transaction arrives that is valid in both eras; the node tags it with Y, its current selection's era. - The node switches to a (denser) selection that is still in era X. - This causes the Mempool to discard validated transactions tagged with a "future era", including Y. If the user could declare the tx's era tag (as they do with today's cardano-cli), then they could declare X, and so the tx could have survived the node's brief reversion to era X.

---

Edit:

Another option would be for blocks and headers to be tagged with the protocol version, but transactions to be tagged with a bespoke transaction version. The developers would increment this version anytime a new protocol version involves changes to the meaning the meaning of well-formed transactions of the previous transaction version. This eliminates the risk of the user submitting a transaction that is well formed in multiple protocol versions but has different semantics.

TODO to what extent is the era tag different from such a transaction version? Maybe this is what @lehins meant before and it's taken me a while to grok it.

[lehins]

@nfrisby I don't fully understand why transactions would include any such tag? As far as ledger is concerned a transaction either deserializes and validates, or it doesn't. So, there should be no surprise to a user. Also, more often than not transactions from previous protocol version will be accepted just fine for the next protocol version, whether it was an intra-era or cross-era hard fork. That is why today translation of transactions in the mempool works when we transition from one era to another.

This backwards compatibility is a nice property, because users could submit transactions and not worry about era transitions and HFCs, therefore it is unclear to me why during transaction submission today there is information about the era that the transaction was built for.

We, in ledger, when we change transaction serialization, we try to do it in the most graceful matter possible for any one era transition, in order to make the transition to the new era smooth. The only reason why we would ever want to change serialization during an intra era hard fork would be to patch some bug, which would have no affect on normal legitimate transactions.

If there is a desire from users to specify validity for protocol version, maybe we could provide ability for a user to supply an optional range of protocol versions that the transaction should be valid for. Considering such envelope information today wouldn't be signed it would be more of UX improvement/restriction, rather than some sort of guarantee, unless in the future era we make this protocol version range part of the TxBody, similarly to validity interval.

How does this scenario work today?

• The node's selection advances from era X to era Y.
• An untagged transaction arrives that is valid in both eras; the node tags it with Y, its current selection's era.
• The node switches to a (denser) selection that is still in era X.
• This causes the Mempool to discard validated transactions tagged with a "future era", including Y.

Cause I'd assume when a node advances from era X to Y all transactions in the mempool are upgraded to that new era Y? If there is a selection of another chain that is still in era X then transactions in the mempool cannot be downgraded to the era X from era Y, at least there is no such mechanism provided by the Ledger. Is the whole mempool reconstructed by deserializing all transactions in era X from their binary representation or how does it work?

[jasagredo]

@lehins

As far as ledger is concerned a transaction either deserializes and validates, or it doesn't.

Does this mean that transactions are also forward compatible in serialization? i.e. a transaction created in Shelley can be deserialized just fine as a Conway transaction? is there no special logic used when upgrading Transactions to later eras?

Or maybe I'm misunderstanding your comment here.

[amesgen]

How does this scenario work today?

• The node's selection advances from era X to era Y.
• An untagged transaction arrives that is valid in both eras; the node tags it with Y, its current selection's era.
• The node switches to a (denser) selection that is still in era X.
• This causes the Mempool to discard validated transactions tagged with a "future era", including Y.

Cause I'd assume when a node advances from era X to Y all transactions in the mempool are upgraded to that new era Y? If there is a selection of another chain that is still in era X then transactions in the mempool cannot be downgraded to the era X from era Y, at least there is no such mechanism provided by the Ledger. Is the whole mempool reconstructed by deserializing all transactions in era X from their binary representation or how does it work?

What is currently happening is that transaction in the mempool always are kept in the era that they were originally in. E.g. if you submit a Shelley transaction today on mainnet, it will be a Shelley transaction in the mempool. Every time we need to do something with it in the context of a specific ledger state (validate/revalidate), we will first, if possible, translate it into the corresponding era of the ledger state (today: Conway), and then invoke the ledger, but we don't modify the tx in the mempool. This way, we do not need to "un-translate" txs in this scenario.

[lehins]

Does this mean that transactions are also forward compatible in serialization? i.e. a transaction created in Shelley can be deserialized just fine as a Conway transaction?

Yes, most of them will. There were some bug fixes and some aspects of a transaction since Shelley have been deprecated (like MIR Certs, etc), but majority of transactions will deserialize in Conway.

This is not always going to be true. The goal is to keep as much backwards compatibility from one era to the next one that follows, but there is no real need to support all eras all the way down to Shelley.

is there no special logic used when upgrading Transactions to later eras?

Nope. Logic is take the bytes and deserialize that transaction in an era that you'd like to validate it in.

[jasagredo]

Does this mean that transactions are also forward compatible in serialization? i.e. a transaction created in Shelley can be deserialized just fine as a Conway transaction?

Yes, most of them will. There were some bug fixes and some aspects of a transaction since Shelley have been deprecated (like MIR Certs, etc), but majority of transactions will deserialize in Conway.

This is not always going to be true. The goal is to keep as much backwards compatibility from one era to the next one that follows, but there is no real need to support all eras all the way down to Shelley.

is there no special logic used when upgrading Transactions to later eras?

Nope. Logic is take the bytes and deserialize that transaction in an era that you'd like to validate it in.

Say I get one of those deprecated MIR certs transactions, which I guess must be regarded as a Shelley transaction. It will succeed to deserialize as a Shelley transaction, and what happens then if I try to translate it to Conway? will it fail to be upgraded? will it be upgraded to something which yields always invalid?

What I am looking to clarify is:

For transaction created in old era with bytes T_bytes, which get deserialized in such era as T_old, translating T_old to the latest era yields transaction T_new, then deserializing T_bytes as if the transaction was already on the latest era also yields transaction T_new.

And also:

FFor transaction created in old era with bytes T_bytes, which get deserialized in such era as T_old, deserializing T_bytes as if it was on the latest era yields an error, translating T_old to the latest era will also yield that same error.

Are these two statements true?

Edit: please consider writing a reply to this comment in this thread I made: https://github.com/IntersectMBO/ouroboros-consensus/pull/1401/files#r1993639143

[coot]

In the future, it would be nice to expand this document and describe the relation between protocol version, BlockNodeToNodeVersion and NodeToNodeVersion.

[lehins]

@jasagredo It is much simpler than you think. When you look at CDDL of a transaction, there is no indication of which era it was built for. Sometimes it can be inferred, eg. if a transaction contains votes than it surely was build for Conway.

So, the way translation works in ledger is we take the bytes of the transaction and try to deserialize it in the new era. In other words, there is no "translation" happens. We simply deserialize the submitted bytes in the era that it needs to be validated in and we retain those bytes. If that transaction is incompatible with this era you will get deserialization error.

The overlap of compatibility of transactions for all eras can be inferred form the CDDL specification. If two different eras have the same specification for the parts of the transaction, then it will deserialize for those two eras. Whether it validate or not that is a different story, because there are plenty of things that can affect validity of a transaction, eg. if cost models protocol parameter were updated then it would effectively invalidate all transactions that use older cost model for the plutus version that was updated.