ouroboros-consensus
ouroboros-consensus copied to clipboard
IntersectMBO
[BUG] - local tx submission is not backward compatible
External
Summary I am using the local transaction submission mini protocol, while submitting a transaction that is built using Alonzo spec, and being submitted with era number 5 (Babbage), the node disconnects the connection and does not return an error message as well.
Steps to reproduce
Build an Alonzo era tx,
Prepare payload with era as 5
Submit using the local transaction submission mini protocol
Expected behaviour Since Babbage era tx spec is Alonzo compatible, a tx built using Alonzo only spec is a valid Babbage era tx. The protocol and node should accept the tx.
Generally, there are no guarantees that the tx binary format evolves in a backwards-compatible manner from era to era, as was the case from e.g. Byron to Shelley. So there is no general guarantee that submitting the same bytes will work with era index i+1 when it worked with i.
On the other hand, Ledger does not randomly change the format, so a serialized Alonzo tx might well be also a valid Babbage tx. Can you maybe post the CBOR of the tx you are trying to submit as a Babbage tx?
@amesgen I find your comment contradicting your own comment. Though I explain the issue in the description and is what you describe in your first paragraph, but it should work as per your second paragraph. Below the tx cbor I am submitting with era 5
84a400828258203a79a6a834e7779c67b0b3cbd3b7271883bbbeac15b1a89d78f057edc25e000b008258203a79a6a834e7779c67b0b3cbd3b7271883bbbeac15b1a89d78f057edc25e000b010182825839007d5a2560d23c3443b98d84c57b0c491311da4b3098de1945c7bcfc4c63ea8c5404f9ed9ae80d95b5544857b2011e3f26b63ddc3be1abd42d1a001e84808258390009ecea977429fa7a4993bc045ea618f3697e6b8eac9d5ea68bba7e4b63ea8c5404f9ed9ae80d95b5544857b2011e3f26b63ddc3be1abd42d821a560ea01ca4581c47be64fcc8a7fe5321b976282ce4e43e4d29015f6613cfabcea28eaba244546573741a3b97c0aa51576f52456d706972654c696368303037391a3443f4a0581c4cd2ea369880853541c5f446725f3e4ecaf141635f0c56c43104923ba14574464c41431b0de0b6b346d4b018581c85ef026c7da6a91f7acc1e662c50301bcce79eb401a3217690aa7044a14574464c41431b000000022eaca140581c92bd3be92d6a6eadd7c01ce9ff485809f3f2eb36845cd7a25c9177bfa14b546f20746865206d6f6f6e01021a0002b9b5031a05a18ef7a100818258202726733baa5c15d8d856c8d94e7d83bcfc7f5661ec7f952f052f311a2443feb258405f9d3d8a703baf700a3015994a3e8702fd7fe2e25d640487944b32ea999f36b314be9674be09b8b8f2c678976ecf994c83086180e854120d81243476c2b89e05f5f6
It should generally work as all previous era tx is a valid Babbage era tx as per the spec. Note that, it does not just return an error, but disconnects the connection, which generally happens when you are communicating with a bad format. But it works well if I use era as 4
cc @JaredCorduan
I don't think there is a contradiction, backwards-compatibility is not guaranteed in general (Byron-to-Shelley is the counterexample), but at the same time, it can be satisfied for e.g. Alonzo-to-Babbage.
I just tested that deserializing your transaction (after wrapping it appropriately) can be deserialized using the code in this repo both with era index 4 and 5, i.e. for 5, prepending 8205d818590210 in hex, corresponding to the CBOR
82 # array(2)
05 # unsigned(5)
d8 18 # tag(24)
59 0210 # bytes(528)
works fine; so the error must be somewhere else.
Pretty-printed deserialized transaction
HardForkGenTx
{ getHardForkGenTx = S
( S
( S
( S
( S
( Z AlonzoTx
{ body = TxBodyConstr BabbageTxBodyRaw
{ btbrSpendInputs = fromList
[ TxIn
( TxId
{ unTxId = SafeHash "3a79a6a834e7779c67b0b3cbd3b7271883bbbeac15b1a89d78f057edc25e000b" }
)
( TxIx 0 )
, TxIn
( TxId
{ unTxId = SafeHash "3a79a6a834e7779c67b0b3cbd3b7271883bbbeac15b1a89d78f057edc25e000b" }
)
( TxIx 1 )
]
, btbrCollateralInputs = fromList []
, btbrReferenceInputs = fromList []
, btbrOutputs = StrictSeq
{ fromStrict = fromList
[ Sized
{ sizedValue =
( Addr Testnet
( KeyHashObj
( KeyHash "7d5a2560d23c3443b98d84c57b0c491311da4b3098de1945c7bcfc4c" )
)
( StakeRefBase
( KeyHashObj
( KeyHash "63ea8c5404f9ed9ae80d95b5544857b2011e3f26b63ddc3be1abd42d" )
)
)
, MaryValue 2000000
( MultiAsset
( fromList [] )
)
, NoDatum
, SNothing
)
, sizedSize = 65
}
, Sized
{ sizedValue =
( Addr Testnet
( KeyHashObj
( KeyHash "09ecea977429fa7a4993bc045ea618f3697e6b8eac9d5ea68bba7e4b" )
)
( StakeRefBase
( KeyHashObj
( KeyHash "63ea8c5404f9ed9ae80d95b5544857b2011e3f26b63ddc3be1abd42d" )
)
)
, MaryValue 1443799068
( MultiAsset
( fromList
[
( PolicyID
{ policyID = ScriptHash "47be64fcc8a7fe5321b976282ce4e43e4d29015f6613cfabcea28eab" }
, fromList
[
( "54657374"
, 999801002
)
,
( "576f52456d706972654c69636830303739"
, 876868768
)
]
)
,
( PolicyID
{ policyID = ScriptHash "4cd2ea369880853541c5f446725f3e4ecaf141635f0c56c43104923b" }
, fromList
[
( "74464c4143"
, 999999998379995160
)
]
)
,
( PolicyID
{ policyID = ScriptHash "85ef026c7da6a91f7acc1e662c50301bcce79eb401a3217690aa7044" }
, fromList
[
( "74464c4143"
, 9373000000
)
]
)
,
( PolicyID
{ policyID = ScriptHash "92bd3be92d6a6eadd7c01ce9ff485809f3f2eb36845cd7a25c9177bf" }
, fromList
[
( "546f20746865206d6f6f6e"
, 1
)
]
)
]
)
)
, NoDatum
, SNothing
)
, sizedSize = 267
}
]
}
, btbrCollateralReturn = SNothing
, btbrTotalCollateral = SNothing
, btbrCerts = StrictSeq
{ fromStrict = fromList [] }
, btbrWithdrawals = Withdrawals
{ unWithdrawals = fromList [] }
, btbrTxFee = Coin 178613
, btbrValidityInterval = ValidityInterval
{ invalidBefore = SNothing
, invalidHereafter = SJust
( SlotNo 94473975 )
}
, btbrUpdate = SNothing
, btbrReqSignerHashes = fromList []
, btbrMint = MultiAsset
( fromList [] )
, btbrScriptIntegrityHash = SNothing
, btbrAuxDataHash = SNothing
, btbrTxNetworkId = SNothing
}
( blake2b_256: SafeHash "8c3414268c4a22996ed9fa410d74567673e587d3c356fd9ae9f2f7a20584a89c" )
, wits = AlonzoTxWitsRaw
{ atwrAddrTxWits = fromList
[ WitVKeyInternal
{ wvkKey = VKey
( VerKeyEd25519DSIGN "2726733baa5c15d8d856c8d94e7d83bcfc7f5661ec7f952f052f311a2443feb2" )
, wvkSig = SignedDSIGN
( SigEd25519DSIGN "5f9d3d8a703baf700a3015994a3e8702fd7fe2e25d640487944b32ea999f36b314be9674be09b8b8f2c678976ecf994c83086180e854120d81243476c2b89e05" )
, wvkKeyHash = KeyHash "ea00161ec2547143255d715d7e3c5cb5d906e6d52bf3b5b866877b65"
, wvkBytes = "\x82X '&s;ª\\x15ØØVÈÙN}\x83¼ü\x7fVaì\x7f\x95/\x5/1\x1a$Cþ²X@_\x9d=\x8ap;¯p
0\x15\x99J>\x87\x2ý\x7fââ]d\x4\x87\x94K2ê\x99\x9f6³\x14¾\x96t¾\x9¸¸òÆx\x97nÏ\x99L\x83\x8a\x80èT\x12\xd\x81$4v¸\x9e\x5"
}
]
, atwrBootAddrTxWits = fromList []
, atwrScriptTxWits = fromList []
, atwrDatsTxWits = TxDatsConstr TxDatsRaw
( fromList [] )
( blake2b_256: SafeHash "45b0cfc220ceec5b7c1c62c4d4193d38e4eba48e8815729ce75f9c0ab0e4c1c0" )
, atwrRdmrsTxWits = RedeemersConstr RedeemersRaw
( fromList [] )
( blake2b_256: SafeHash "45b0cfc220ceec5b7c1c62c4d4193d38e4eba48e8815729ce75f9c0ab0e4c1c0" )
}
( blake2b_256: SafeHash "0c463af8d9b51d4c504379c56c41743f47a087e842af9963daf82dc24b9db912" )
, isValid = IsValid True
, auxiliaryData = SNothing
}
)
)
)
)
)
}
HardForkGenTx
{ getHardForkGenTx = S
( S
( S
( S
( Z AlonzoTx
{ body = TxBodyConstr AlonzoTxBodyRaw
{ atbrInputs = fromList
[ TxIn
( TxId
{ unTxId = SafeHash "3a79a6a834e7779c67b0b3cbd3b7271883bbbeac15b1a89d78f057edc25e000b" }
)
( TxIx 0 )
, TxIn
( TxId
{ unTxId = SafeHash "3a79a6a834e7779c67b0b3cbd3b7271883bbbeac15b1a89d78f057edc25e000b" }
)
( TxIx 1 )
]
, atbrCollateral = fromList []
, atbrOutputs = StrictSeq
{ fromStrict = fromList
[
( Addr Testnet
( KeyHashObj
( KeyHash "7d5a2560d23c3443b98d84c57b0c491311da4b3098de1945c7bcfc4c" )
)
( StakeRefBase
( KeyHashObj
( KeyHash "63ea8c5404f9ed9ae80d95b5544857b2011e3f26b63ddc3be1abd42d" )
)
)
, MaryValue 2000000
( MultiAsset
( fromList [] )
)
, SNothing
)
,
( Addr Testnet
( KeyHashObj
( KeyHash "09ecea977429fa7a4993bc045ea618f3697e6b8eac9d5ea68bba7e4b" )
)
( StakeRefBase
( KeyHashObj
( KeyHash "63ea8c5404f9ed9ae80d95b5544857b2011e3f26b63ddc3be1abd42d" )
)
)
, MaryValue 1443799068
( MultiAsset
( fromList
[
( PolicyID
{ policyID = ScriptHash "47be64fcc8a7fe5321b976282ce4e43e4d29015f6613cfabcea28eab" }
, fromList
[
( "54657374"
, 999801002
)
,
( "576f52456d706972654c69636830303739"
, 876868768
)
]
)
,
( PolicyID
{ policyID = ScriptHash "4cd2ea369880853541c5f446725f3e4ecaf141635f0c56c43104923b" }
, fromList
[
( "74464c4143"
, 999999998379995160
)
]
)
,
( PolicyID
{ policyID = ScriptHash "85ef026c7da6a91f7acc1e662c50301bcce79eb401a3217690aa7044" }
, fromList
[
( "74464c4143"
, 9373000000
)
]
)
,
( PolicyID
{ policyID = ScriptHash "92bd3be92d6a6eadd7c01ce9ff485809f3f2eb36845cd7a25c9177bf" }
, fromList
[
( "546f20746865206d6f6f6e"
, 1
)
]
)
]
)
)
, SNothing
)
]
}
, atbrCerts = StrictSeq
{ fromStrict = fromList [] }
, atbrWithdrawals = Withdrawals
{ unWithdrawals = fromList [] }
, atbrTxFee = Coin 178613
, atbrValidityInterval = ValidityInterval
{ invalidBefore = SNothing
, invalidHereafter = SJust
( SlotNo 94473975 )
}
, atbrUpdate = SNothing
, atbrReqSignerHashes = fromList []
, atbrMint = MultiAsset
( fromList [] )
, atbrScriptIntegrityHash = SNothing
, atbrAuxDataHash = SNothing
, atbrTxNetworkId = SNothing
}
( blake2b_256: SafeHash "8c3414268c4a22996ed9fa410d74567673e587d3c356fd9ae9f2f7a20584a89c" )
, wits = AlonzoTxWitsRaw
{ atwrAddrTxWits = fromList
[ WitVKeyInternal
{ wvkKey = VKey
( VerKeyEd25519DSIGN "2726733baa5c15d8d856c8d94e7d83bcfc7f5661ec7f952f052f311a2443feb2" )
, wvkSig = SignedDSIGN
( SigEd25519DSIGN "5f9d3d8a703baf700a3015994a3e8702fd7fe2e25d640487944b32ea999f36b314be9674be09b8b8f2c678976ecf994c83086180e854120d81243476c2b89e05" )
, wvkKeyHash = KeyHash "ea00161ec2547143255d715d7e3c5cb5d906e6d52bf3b5b866877b65"
, wvkBytes = "\x82X '&s;ª\\x15ØØVÈÙN}\x83¼ü\x7fVaì\x7f\x95/\x5/1\x1a$Cþ²X@_\x9d=\x8ap;¯p
0\x15\x99J>\x87\x2ý\x7fââ]d\x4\x87\x94K2ê\x99\x9f6³\x14¾\x96t¾\x9¸¸òÆx\x97nÏ\x99L\x83\x8a\x80èT\x12\xd\x81$4v¸\x9e\x5"
}
]
, atwrBootAddrTxWits = fromList []
, atwrScriptTxWits = fromList []
, atwrDatsTxWits = TxDatsConstr TxDatsRaw
( fromList [] )
( blake2b_256: SafeHash "45b0cfc220ceec5b7c1c62c4d4193d38e4eba48e8815729ce75f9c0ab0e4c1c0" )
, atwrRdmrsTxWits = RedeemersConstr RedeemersRaw
( fromList [] )
( blake2b_256: SafeHash "45b0cfc220ceec5b7c1c62c4d4193d38e4eba48e8815729ce75f9c0ab0e4c1c0" )
}
( blake2b_256: SafeHash "0c463af8d9b51d4c504379c56c41743f47a087e842af9963daf82dc24b9db912" )
, isValid = IsValid True
, auxiliaryData = SNothing
}
)
)
)
)
}
I can understand that the deserializing works, as it should. Does this mean the problem lies with Ouroboros Network? Because submitting this tx works only with era 4, and not 5 via the mini protocol.
There were no protocol level changes since a very long time, so I'd be quite surprised if the problem lives in ouroboros-network. What error do you get when you submit the tx? Does the local-tx-submission mini-protocol returns without an error? Are you using cardano-cli or some 3rd party tool for submitting the tx?
if we use cardano-cli, it does not require to define the era hence it calculates the correct era and submits the tx thus works fine.
I am using the local tx submit mini protocol directly as detailed in the description, hence I now have to define the tx era myself and that is how this issue is found. If the code is not changed for a long time, possible the issue is present from a long time because so far only IOG tools submit the tx, and some other community tools which submit the tx figures out the era manually and define it before submitting (as described if the correct era is defined it works)
If we submit an alonzo era tx in babbage era, with era number as babbage (5), the node just disconnects the connection without an error msg ofc. Note as described in the previous messages, if we submit the same tx with era as 4 while in Babbage era it works fine.
~What protocol error is returned by the responder side of the local-tx-submission mini-protocol?~
Oh, I see there's no message back. Is there anything in the cardano-node logs? It should log the exception thrown by the mini-protocol.
For my own tools, this is the work-around I have to do to ensure I'm submitting the right era for a given transaction. It feels fragile and prone to bugs even though it currently works. If I submit a wrong era, like @ashisherc said, you just get dropped by the node and disconnected with no error message.
internal val era: Int by lazy {
try {
val txArray = CborObject.createFromCborByteArray(signedTxCborByteArray) as CborArray
if (txArray.elementAt(txArray.size() - 2) is CborSimple) {
// alonzo, babbage, or conway
val txBody = txArray.elementAt(0) as CborMap
val txDestsArray = txBody[TX_DESTS_INDEX] as CborArray
val txCerts = txBody[TX_CERTS_INDEX] as? CborArray
val witnessSet = txArray.elementAt(1) as CborMap
val auxData = txArray.elementAt(3) as? CborMap
if (
containsSetTag(txBody) ||
containsSetTag(witnessSet) ||
containsGovernanceAction(txCerts) ||
containsConwayTxKeys(txBody) ||
containsConwayWitnessSet(witnessSet) ||
containsConwayAuxData(auxData)
) {
log.debug { "Submitting CONWAY tx" }
CardanoEra.CONWAY.eraNumber
} else if ((!txDestsArray.isEmpty && (txDestsArray.any { it is CborMap })) ||
(txBody[TX_COLLAT_RETURN_INDEX] != null) ||
(txBody[TX_COLLAT_TOTAL_INDEX] != null) ||
(txBody[TX_REFERENCE_INPUTS_INDEX] != null)
) {
// in Babbage, any of the out UTXOs could be a CborMap or it's using one of the new features
log.debug { "Submitting BABBAGE tx" }
CardanoEra.BABBAGE.eraNumber
} else {
log.debug { "Submitting ALONZO tx" }
CardanoEra.ALONZO.eraNumber
}
} else {
log.debug { "Submitting MARY tx" }
CardanoEra.MARY.eraNumber
}
} catch (e: Throwable) {
log.error(e) { "Error decoding MsgSubmitTx!" }
CardanoEra.MARY.eraNumber
}
}
We could not reproduce this in a local (Consensus) test. In future releases, we will enable certain tracers by default, which include the tracers that log exceptions that cause local client disconnection. In the meantime could you set the severity of the local tx submission protocol to Debug?
"mapSeverity": {
"cardano.node.LocalTxSubmissionProtocol": "Debug"
}
@dnadales no logs even after the suggested config change. the connection is just dropped.
@ashisherc @AndrewWestberg Do you still have repros for this Issue, 1.5 years later 😬?
The only element that's coming to my mind but is not discussed above is what handshake version was being used for the NTC connection. However (ruling out bugs that would be surprising, eg related to the enable-experimental-eras flags), you wouldn't have been able to submit any Babbage txs if your negotiated version was stale.
no logs even after the suggested config change. the connection is just dropped.
I agree that not having the log message for this is very painful.
My guess is that this is the exception that we're all surmising is being thrown https://github.com/IntersectMBO/ouroboros-network/blob/28b731cde005a1de0b4be0ea0bd16852e827c1bc/ouroboros-network-framework/src/Ouroboros/Network/Driver/Simple.hs#L152 --- I don't know why it was not appearing in the node log.
FWIW, I just synced today's node on preprod, and then used a hacked up db-analyser to decode the bytestring from @amesgen 's comment above with the Babbage 5 era tag into a GenTx (CardanoBlock StandardCrypto) using the decodeNodeToClient method with CardanoNodeToClientVersion15 and then successfully applied that tx with applyTx to the ledger state from preprod slot 3011839, which is immediately after the block that introduced the txouts that @ashisherc 's example bytestring above consumes.
https://preprod.cexplorer.io/tx/3a79a6a834e7779c67b0b3cbd3b7271883bbbeac15b1a89d78f057edc25e000b
So: not only do those bytes decode in Babbage using the methods we think are involved, but the resulting Babbage tx is also valid in the first possible ledger state it could have been valid in just considering its TxIns.
Caveats:
- I used today's
mainbranch ofdb-analyser: ie ebcbe32fa. - I specified the latest NTC version.
- I used the config files from https://book.play.dev.cardano.org/env-preprod.html
- I used the
decodeNodeToClientmethod directly. - I
showed the resulting ledger state, forcing all thunks.
Generally, there are no guarantees that the tx binary format evolves in a backwards-compatible manner from era to era
There is even no guarantee from one epoch to another. All it takes is a protocol parameter update and a previously valid transaction can become invalid.
at the same time, it can be satisfied for e.g. Alonzo-to-Babbage.
More often than not transaction built for previous era will still work for the next era. So I really don't understand why consensus submission protocol requires this era tagging, makes no sense to me.
More often than not transaction built for previous era will still work for the next era. So I really don't understand why consensus submission protocol requires this era tagging, makes no sense to me.
THIS exactly. We are using a very similar snippet as shared by @AndrewWestberg to figure a correct era tag to be able to submit tx and it has made tooling very difficult.
Generally, there are no guarantees that the tx binary format evolves in a backwards-compatible manner from era to era
There is even no guarantee from one epoch to another. All it takes is a protocol parameter update and a previously valid transaction can become invalid.
Backwards-compatibility for the tx binary format and for general tx validity (with the latter depending on the protocol parameters) are different things, right?
at the same time, it can be satisfied for e.g. Alonzo-to-Babbage.
More often than not transaction built for previous era will still work for the next era. So I really don't understand why consensus submission protocol requires this era tagging, makes no sense to me.
(I think) the historical reason for this is the fact that transactions between Byron and Shelley were completely different, so it made sense to indicate this as a tag (in combination with the tag that basically every entity (blocks/headers/etc.) has such a tag today). But indeed, it might be nice to change this, as has recently been discussed in #1401, albeit for somewhat different reasons. But the better user experience is definitely a factor!
Backwards-compatibility for the tx binary format and for general tx validity (with the latter depending on the protocol parameters) are different things, right?
Successful deserialization is a prerequisite of a valid transaction. But, yes, they are different things.
I had to yet again, update my comment above. I had missed an edge case where somebody puts cbor tag 258 only in the witness set which makes it a conway transaction. This happens sometimes when the DApp building the tx makes it for an earlier era, but the wallet signs with conway by default.
The node really needs to take care of this for us. I want to send valid cbor tx bytes and if it's valid in any era, it just works. Very painful to continue to maintain this code and finding random new edge cases.
The result of this era mismatch if using local tx submission miniprotocol is that the node just silently drops the connection and everything after it fails until a new handshake is made on a new unix socket. No error messages at all about bad era, etc... The connection is simply dropped.
@AndrewWestberg, we should send the reason why the connection is dropped to the client, and it should show it in a human-readable form. Currently, one needs to examine node logs, find the right place, and quite possibly the reason is not human-readable.
@coot this is a step in the right direction, but I would prefer to not have to submit an era with the transaction and get dropped if I guess wrong.
The node already processes txns from any era if it arrives in a block not knowing the era. Why do I have to supply the era to the node when submitting a tx?
Sure, I am just noting how network could help to diagnose issues with invalid txs.
@ch1bo So is this "not" happening? I would still very much like to see it in a release.
@AndrewWestberg This was indexed by some automation @RSoulatIOHK created and I just removed it because it is not related to Leios. These additions and removals to the Leios board bear no meaning in whether this will be addressed or not.
