Caravan export

[notmandatory]

Description

This is a new module for importing and exporting a wallet descriptor and network from and to a Caravan config JSON file.

Notes to the reviewers

I only have minimal happy path tests now. More negative test cases are needed.

Open issues:

• [ ] What is the client.type field? are there enum values?
• [ ] Is it OK to put the descriptor key hash in the .extendedPublicKeys[].name fields when exporting?
• [ ] I'm only supporting "sortedmulti" inner expressions, are any others expected?
• [ ] Change descriptors aren't supported but could be, need to confirm derivation path
• [x] Add tests for all example configs
• [x] Do private key wallets/vaults need to be supported? No.

Checklists

All Submissions:

• [x] I've signed all my commits
• [x] I followed the contribution guidelines
• [x] I ran cargo fmt and cargo clippy before committing

New Features:

• [x] I've added tests for the new feature
• [x] I've added docs for the new feature
• [x] I've updated CHANGELOG.md

Bugfixes:

• [ ] This pull request breaks the existing API
• [ ] I've added tests to reproduce the issue which are now passing
• [ ] I'm linking the issue being fixed by this PR

[notmandatory]

@bucko13 here's a first cut at a import/export tool. Happy to walk through it with you some time. I have a few open questions, see above.

[afilini]

As a sidenote: the way this is coded it only supports importing/exporting public keys. Is this the goal of caravan, giving you a watch-only descriptor?

If you want to support private keys as well it get a bit more complicated, but still very doable!

[notmandatory]

Thanks for the suggestions! will get those fixed. AFAIK caravan only supports pub keys but will add that to my list of issues to get an answer on.

[bucko13]

🎉 exciting to see progress on this!

One general comment that will probably impact the further development of this is that I think it might be easier, especially for long term maintenance, to keep this as general as possible and maybe not even Caravan specific at all. One of the primary things that I think is missing from the space is a way to effectively parse and encode descriptors in JavaScript based applications. And to do it in a way that is relying on the canonical implementations of the descriptor/miniscript tooling would be ideal so we know we have all the guarantees that the reference implementation(s) provide (without having to maintain).

The primary use case I envision for bdk bindings in wasm is that we would be able to feed some function a descriptor string (maybe even support for more complex policies for future extensibility, but really multisig descriptors is all Caravan and nearly every other coordinator in the space really cares about) to a function and get a parseable object/JSON of the Extended Key Derivation/Origin information (XKey from bdk kind of I think? There are so many names for this amalgamation of information). The rest, such as client information, is totally ancillary and implementation specific. If you think about where this would be used in Caravan, it would be in the main wallet loading area. A descriptor string text input could supersede most of the fields on this screen, and that's what I think would be most useful for bdk to handle.

The nice thing about generalizing it this way too is that your team doesn't have to try to keep up maintainability with whatever the caravan config may or may not need to support. So considering the screenshot above, a possible user flow would look like:

1. input descriptor string(s) (could be one if BIP 88 style), into a text input at the top of the window
2. in the background bdk wasm bindings would: i) validate descriptor(s) ii) pull out all relevant wallet information
3. the UI might then prefill all relevant details it can pull from the descriptor: xpubs, network (?), quorum size
4. user would fill out any other extraneous information that caravan might care about like client info and name

The nice thing about this too imo is that it could be used as a common interface to convert to any other wallet config pretty easily once we have this to/from descriptor api, e.g. Descriptor -> Caravan -> Coldcard. The final step could be facilitated completely in the JS side with something that looks like this:

const coldcardConfig = '...'
const wallet = ColdcardConfig.deserialize(coldcardConfig)

wallet.getDescriptors() // ["wsh(sortedmulti(2,[3h8280]xpub.../0/*, [748d8s0]xpub.../0/*))", "..."]
const descriptor = wallet.toBip88Descriptor() // "wsh(sortedmulti(2,[3h8280]xpub.../{0,1}/*, [748d8s0]xpub.../{0,1}/*))"

const carvanConfig = CaravanWalletConfig.fromBip88Descriptor(
		descriptor,
		{ 
			...wallet, // wallet.name, wallet.uuid, wallet.network
			startingAddressIndex: 10 
		} 
)

caravanConfig.serialize() // { ... } -> caravan compatible json
wallet.serialize() // "..." -> coldcard wallet config text

// these should be the same and export standard descriptors
// that can be imported to bitcoind (for example)
assert_equal(caravanConfig.getDescriptors(), wallet.getDescriptors())

Under the hood, the getDescriptors and toBip88Descriptors would just be using the bdk utilities. Everything else is just implementation detail. (the bip88 stuff would just expand the bip88 {0,1} style wildcards to two separate descriptor strings).

Anyway, that's just how I've been thinking about it and I feel like should make your work easier.

For priv keys: yes, caravan itself will only ever be watch only as all signing should be delegated out to some external signer. Currently we support Coldcard, trezor, ledger, and manual. We intend on adding bcr2 QR support soon which would open up support for many other signers. That said, given the approach described above of this being a generalized tool, there's no reason this couldn't support priv keys and then in other applications that might need it and want that capability, it would be there (I'll probably build these tools with the bindings into unchained-bitcoin to use in Caravan, but also make available to other applications that wish to use it). It's also not an urgent requirement for our goals at least.

(other questions answered inline)

[notmandatory]

For the larger discussion about using BDK for generic parsing of descriptors for use in a TypeScript app I think we can do this with the existing (but still experimental) descriptor::policy module. This module will parse any descriptor into a common Policy structure that can be easily serialized to JSON. I originally intended to use the Policy parser for this PR but since the Caravan config is so constrained it was easier to pull what I needed directly from the Descriptor structure.

If you want to see what the Policy JSON output looks like you can use the bdk-cli tool to parse any wallet descriptor. Installation instruction are here. Once installed the command to parse a descriptor is:

bdk-cli wallet -d $ALICE_DESCRIPTOR policies

If this looks like something you can use I'll make an example for doing the same parsing from TypeScript using the bdk-wasm pkg.

Ultimately I believe descriptors (with public keys) will be the "lingua franca" for all wallet configs since they contain everything a user needs (besides the master keys) to restore a wallet. Even if you include other ancillary info in your config that other wallets won't understand such as wallet name, mapping user entered names to key fingerprints, the blockchain client configs, etc. the user can still always use the descriptors to monitor and spend using any descriptor based wallet software.

[bucko13]

Ultimately I believe descriptors (with public keys) will be the "lingua franca" for all wallet configs since they contain everything a user needs (besides the master keys) to restore a wallet. Even if you include other ancillary info in your config that other wallets won't understand such as wallet name, mapping user entered names to key fingerprints, the blockchain client configs, etc. the user can still always use the descriptors to monitor and spend using any descriptor based wallet software.

Yep, this is exactly my thinking and what my larger goal for using this tool would be.

So, I tested out bdk-cli locally:

$ ➜ ALICE="wsh(sortedmulti(1,[9d120b19/48'/0'/0'/2']xpub6FDrnnUsgQSwRFazYbVDs9eadQaNV13f5dtQDoWrCuMNq2qgMH7GevctMAm3PeHq3KBkh9BgA8iPfaHYACHFpfueYdeAUtjjEH3vMJWEKfu,[5c9e228d/48'/0'/0'/2']xpub6EgGHjcvovyN3nK921zAGPfuB41cJXkYRdt3tLGmiMyvbgHpss4X1eRZwShbEBb1znz2e2bCkCED87QZpin3sSYKbmCzQ9Sc7LaV98ngdeX))#dh7v9p4x"
$ ➜ bdk-cli -n bitcoin wallet -d $ALICE policies 

With the following return:

{
  "external": {
    "contribution": {
      "items": [],
      "m": 1,
      "n": 2,
      "sorted": true,
      "type": "PARTIAL"
    },
    "id": "hw3hgusf",
    "keys": [
      {
        "fingerprint": "9d120b19"
      },
      {
        "fingerprint": "5c9e228d"
      }
    ],
    "satisfaction": {
      "items": [],
      "m": 1,
      "n": 2,
      "sorted": true,
      "type": "PARTIAL"
    },
    "threshold": 1,
    "type": "MULTISIG"
  },
  "internal": null
}

Information it looks like we do get from policies is:

• m (required signers) and n (total signers)
• sorted and multi
• xfps for the keys involved

Unless there's something in my command that I'm missing (didn't see any additional options in the help though), info we'd still need returned:

• checksums
• key origins (path, extended public key, as well as the xfp)
• script type

inferring the network from the xpub version as well would be nice, but as discussed elsewhere, it's not a guarantee (could let that be the responsibility of the consuming application to validate or allow to be overwritten).

It looks like most of this work is being done in this PR by the parse_descriptor function

[notmandatory]

Unless there's something in my command that I'm missing (didn't see any additional options in the help though), info we'd still need returned:

• checksums
• key origins (path, extended public key, as well as the xfp)
• script type

No there's no other options right now to get the above extra info. But I agree if we added that info somewhere in the Policy structure it would be generally useful for parsing a Descriptor and displaying it's spending requirements. I created #597 to track this suggestion, but it might take some time to get to as the team is now focused on getting Taproot support added (which also impacts the policy module).

[notmandatory]

@bucko13 I've extracted this code into a new standalone project that uses the new bdk 1.0-alpha as a dependency and is compiled to wasm for use in a web page. I also created a very basic web page example. Everything should work for converting between descriptors and caravan config json. But much docs and cleanup needed before publishing it as a npm package.

https://github.com/notmandatory/caravan-rs https://github.com/notmandatory/caravan-rs-example

[notmandatory]

Very old PR and not part of 1.0 milestone so closing for now. May take it back up post 1.0 as an example or new add-on module.