Serialisation-lib does not parse eUTXO transactions generated by Cardano-cli

[kieransimkin]

We're generating cbor via cardano-cli to be signed by Nami wallet (which uses serialization-lib). Serialization-lib fails to parse eUTXO transactions generated by cardano-cli - in particular it seems to be failing on the witnesses field - I've found this post describing the same problem we're having: https://cardano.stackexchange.com/questions/4336/signing-a-tx-with-nami-wallet It seems to indicate it's the difference between the witness field being [], or {}. This would seem to be corroborated by the fact that if you sign the message with cardano-cli, serialization-lib then parses the transaction correctly (but of course, then you can't sign it with Nami).

This is really problematic for us - we did try manually rewriting the cbor to replace the [] with {} but this was unsuccessful, so we're now having to painstakingly rewrite our transaction generation code to be done in serialization-lib rather than cardano-cli.

[kieransimkin]

Here is some example cbor from Cardano-cli: 86a600828258207a9ae75e073a713791cc3045c2c2f62b925c4d33ae670da445ec80f20c459d7f0082582093ce7d39fc764e8ec9026a50f1f41f0d18649e18378718e91c45060fead0c703010d80018382583900af035fb9a547276bbb86a1936882524463eccb82465fcdf44489013713a39f099c533eb97b848aea0ae05850b4b40ec1ec05541a3f84c9b11a0483d91a83581d70eb18d99e0c392b18a436d595b5223e8fff559c4643118c1d403dfaab821a001e8480a1581c79e78a324e27842aa2ccaa21a710198b1db2c2a835b70d663b68a9dca1534d79546573746e6574547231303030303237310158201db43f14402b617c93f860381ec852799320d35c1be86386dffa53d792d846d582583900af035fb9a547276bbb86a1936882524463eccb82465fcdf44489013713a39f099c533eb97b848aea0ae05850b4b40ec1ec05541a3f84c9b1821a01312d00a2581c6b8d07d69639e9413dd637a1a815a7323c69c86abbafb66dbfdb1aa7a14002581c79e78a324e27842aa2ccaa21a710198b1db2c2a835b70d663b68a9dca2534d79546573746e65745472313030303032363701534d79546573746e65745472313030303032363801021a0002d8a50e8007582053d1932595c943555dd467aa2d3a36efdc865caf35ad77c04c270edcfd2dc39c9fff8080f5d90103a100a119013863626172

This is rejected by serialization lib as being invalid. You can easily verify this yourself if you have Nami installed by opening Chrome on a non-local web page, then opening Chome developer tools, selecting the 'Console' tab and issuing the following commands:

cardano.enable(); cardano.signTx('cborhere');

This is not specific to Nami however, it's the underlying serialization-lib that's emitting the error.

[vsubhuman]

Here is some example cbor from Cardano-cli: 86a600828258207a9ae75e073a713791cc3045c2c2f62b925c4d33ae670da445ec80f20c459d7f0082582093ce7d39fc764e8ec9026a50f1f41f0d18649e18378718e91c45060fead0c703010d80018382583900af035fb9a547276bbb86a1936882524463eccb82465fcdf44489013713a39f099c533eb97b848aea0ae05850b4b40ec1ec05541a3f84c9b11a0483d91a83581d70eb18d99e0c392b18a436d595b5223e8fff559c4643118c1d403dfaab821a001e8480a1581c79e78a324e27842aa2ccaa21a710198b1db2c2a835b70d663b68a9dca1534d79546573746e6574547231303030303237310158201db43f14402b617c93f860381ec852799320d35c1be86386dffa53d792d846d582583900af035fb9a547276bbb86a1936882524463eccb82465fcdf44489013713a39f099c533eb97b848aea0ae05850b4b40ec1ec05541a3f84c9b1821a01312d00a2581c6b8d07d69639e9413dd637a1a815a7323c69c86abbafb66dbfdb1aa7a14002581c79e78a324e27842aa2ccaa21a710198b1db2c2a835b70d663b68a9dca2534d79546573746e65745472313030303032363701534d79546573746e65745472313030303032363801021a0002d8a50e8007582053d1932595c943555dd467aa2d3a36efdc865caf35ad77c04c270edcfd2dc39c9fff8080f5d90103a100a119013863626172

Can confirm, this fails with this error:

DeserializeError { location: Some("Transaction.witness_set.TransactionWitnessSet"), failure: CBOR(Expected(Map, Array)) }

Expected Map, got Array

[kieransimkin]

Alessandro claims that [] is invalid, but I would suggest that since the transaction came out of Cardano-cli, it is, by definition, valid. Either way, since it should be such a trivial fix, is there anything we can do to get this sorted? Like I say, we've tried manually byte-hacking the cbor with no success, but this is really holding us up at a time when we need to get to market ASAP - anything you can do to sort it out for us?

[vsubhuman]

but I would suggest that since the transaction came out of Cardano-cli, it is, by definition, valid

I'm not gonna necessarily argue against supporting a compatibility hack for this at the moment, but I'd be careful about defining the validness of an output by whether the tool is developed by the same company as the actual standard or not )

There's a protocol spec and IOHK tools like any other client might either follow it correctly or not and might have bugs as well.

[vsubhuman]

The protocol spec clearly defines the witness-set as a map where all keys are optional so an empty set must be an empty map: https://github.com/input-output-hk/cardano-ledger/blob/37322df14d44d1370e8a72b677815d64a92baa00/alonzo/test/cddl-files/alonzo.cddl#L252-L259

So a bug can be reported to the cardano-node repo with an example on how to replicate it using the cardano-cli: https://github.com/input-output-hk/cardano-node/issues

[kieransimkin]

There's a protocol spec and IOHK tools like any other client might either follow it correctly or not and might have bugs as well.

That's a very valid point- sorry, my frustration with the issues we've been having, working with bleeding edge technologies is probably coming through a bit and I'm at the point where I want it to Just Work! ;)

Thanks for confirming that for me, I was just about to ask if you could confirm that - I will indeed raise an issue with IOHK. Thanks for your time and I'm sorry for falsely accusing your code of being wrong!

[vsubhuman]

There's a protocol spec and IOHK tools like any other client might either follow it correctly or not and might have bugs as well.

That's a very valid point- sorry, my frustration with the issues we've been having, working with bleeding edge technologies is probably coming through a bit and I'm at the point where I want it to Just Work! ;)

Thanks for confirming that for me, I was just about to ask if you could confirm that - I will indeed raise an issue with IOHK. Thanks for your time and I'm sorry for falsely accusing your code of being wrong!

No probs, mate 👌 Link the issue here once you get it reported and then we'll see from their answer if we wanna add a hack for it on our side or something

[kieransimkin]

Cheers :)

[pyropy]

@kieransimkin Have you found some kinda workaround for this? I'm decoding cbor usingcbor library, and I got array as an result. I can't find where the witness-set is actually set, so I could replace it and then encode transaction again.

[kieransimkin]

@kieransimkin Have you found some kinda workaround for this? I'm decoding cbor usingcbor library, and I got array as an result. I can't find where the witness-set is actually set, so I could replace it and then encode transaction again.

Nope - the bug is with Cardano-cli, I've submitted a bug request to IOHK, which has been open 17 days and hasn't been looked at yet as far as I can tell: https://github.com/input-output-hk/cardano-node/issues/3370 Might help if you go and add your own comment on there?

Our workaround has been to generate the transactions with Serialization-lib rather than Cardano-cli - annoyingly, this involved quite a significant detour for us. Also, there's quite a few weird tweaks and hacks needed to get it working with Serialization-lib too - not a simple drop-in replacement as I had hoped!

We will be documenting the various trials and tribulations we've been through to get our smart contract escrow working on our blog, as it seems there are only a very small number of people in the world who know all the little tricks needed to get smart contracts working on Cardano - those that have managed to get it working, aren't so eager to share the knowledge with their competitors! That's not our approach though - we're going full opensource, and we'll share how we did it. I'm afraid you will have to wait until after our launch before we get around to documenting any of this though :) If you want to join us on our Discord, myself or Torben (our Plutus guru) will be happy to share what we know as far as time permits. http://artifct.app/

[pyropy]

It seems like the issue doesn't stop at transaction witness set, even after trying to hack my way around and setting the witness-set to empty map new issues appears.

Error:

Uncaught (in promise) Deserialization failed in Transaction.body.TransactionBody because: Invalid cbor: not the right type, expected `Map' byte received `Array'.

Workaround code

  const cliTx = '86a80081825820482b507dcc7e0f41e05a87c16875a843940ab95e7aeeb70f7328364a89304ee4010d81825820c163e5a6ad1c2dd258af26b0954034b748b8cb32494def9d378da1378ad9e5fa000181825839001ccda49705cc7fca4b045ebf4c5c86b87910e0bc2d43a21b85f58ce6dd281dff622d735a69da8b141c3911fab6672ae220b72e277a14b2a71a02f83c56021a0002b42a031a02a2128c08000e800b58201d7d190f93685ce1f6b7047684ec522cb2fbdece54675a9f37f73c8652b666d89f8201588c588a0100003332223232323233222225335300a33225335300c0021001100d33350075009003335006500848150ccd401d4024008cd401940212054100513263530043357389211d496e636f727265637420646174756d2e2045787065637465642034322e00005498480048004480044800448dd4000891199ab9a3375e00400200a0082440042440024003ff81182a81840000182a821937bc1a004abd2ff5f6'

  const decodedTx = await cbor.decodeAll(cliTx)

  const witnessSet = new Map()
  
  // spec
  // transaction =
  // [ transaction_body
  // , transaction_witness_set
  // , bool
  // , auxiliary_data / null
  // ]

  const hackedTx = [
    [
      decodedTx[0][0], // transaction_body
      witnessSet, // transaction_witness_set
      decodedTx[0][4], // bool
      decodedTx[0][5], // auxiliary_data
    ]
  ]

[Mercurial]

I also went through this issue and hopefully we can get a more official fix / resolution for this and get all tools aligned in terms of compatibility.

[vsubhuman]

We prolly gonna need to add a compatibility hack for this before releasing 10 then

[pyropy]

What's the exact shape of witness set that ledger/cli produces? I working on a separate data structure that should decode/encode witness set as cli produces it.

I've tried these kind of shapes [{...}] and {[[...]]} but it doesn't seem like that's it.

[vsubhuman]

What's the exact shape of witness set that ledger/cli produces? I working on a separate data structure that should decode/encode witness set as cli produces it.

I've tried these kind of shapes [{...}] and {[[...]]} but it doesn't seem like that's it.

The witness set must be a map with optional numeric keys where values are arrays of specific witness types, see CDDL: https://github.com/input-output-hk/cardano-ledger/blob/37322df14d44d1370e8a72b677815d64a92baa00/alonzo/test/cddl-files/alonzo.cddl#L252-L259

So for a regular transaction with only vkey witnesses it should be something like: {0: [...]}

[pyropy]

What's the exact shape of witness set that ledger/cli produces? I working on a separate data structure that should decode/encode witness set as cli produces it. I've tried these kind of shapes [{...}] and {[[...]]} but it doesn't seem like that's it.

The witness set must be a map with optional numeric keys where values are arrays of specific witness types, see CDDL: https://github.com/input-output-hk/cardano-ledger/blob/37322df14d44d1370e8a72b677815d64a92baa00/alonzo/test/cddl-files/alonzo.cddl#L252-L259

So for a regular transaction with only vkey witnesses it should be something like: {0: [...]}

Yeah but witness set is not actually serialized as per design document by cardano-cli itself, right ? I'm trying to decode/encode witness-set as it's done by cardano-cli :D

[vsubhuman]

Yeah but witness set is not actually serialized as per design document by cardano-cli itself, right ? I'm trying to decode/encode witness-set as it's done by cardano-cli :D

Non-empty sets are serialized correctly, the only problem from what I understood is that when it's empty (not signed yet) then it looks like [] instead of {}

[pyropy]

If you want to load transaction build on backend by CLI in cardano-serialization-lib you can sign it first with some random keys and then load the CBOR. After that you can swap out our witness-set and you are good to go.

[kieransimkin]

We prolly gonna need to add a compatibility hack for this before releasing 10 then

Yes please do, this little bug cost us so much time integrating our smart contracts - it's really holding back a lot of us dApp developers.

[lisicky]

@topcoder1208 Hi! Which line of the code emits the error ?

Also witnessSet.vkeys()?.free(); doesn't clean vkeys inside witnessSet. See https://github.com/Emurgo/cardano-serialization-lib/issues/578