bitcoin
BIP draft: Binary Output Descriptors
A BIP 174 PSBT may contain an extended key for deriving input and output addresses. This document proposes an additional field for PSBTs to represent arbitrary BIP 380 output script descriptors.
To support transfer of output descriptors outside signing flows, the proposal makes the unsigned transaction optional.
I don't think PSBT is the right way to be doing this. Descriptors are not necessary in PSBT, and relaxing the invariant that a PSBT represents a transaction (by allowing a PSBT to include no transaction data) would break many parsers and also just is antithetical to PSBT. It's not a general purpose format for sending data around - you could use protobuf if you wanted that.
If the goal is to have a machine parsable representation of descriptors, then by all means, do that. But it doesn't need to (nor do I think it should) interact with PSBT, other than having a different header that identifies it.
It's also unnecessary to have a descriptor in a PSBT. A PSBT can contain all of the data to derive it from the scriptPubKey at a later time. If an updater has the descriptor in order to put it in the PSBT, then it can also put the rest of the data in the PSBT so that the descriptor can be inferred. I just don't see how this proposal is at all useful.
Also, descriptors are a textual format, and encoding text in binary is just not very efficient. There are better ways to do that if you really wanted to.
I don't think PSBT is the right way to be doing this. Descriptors are not necessary in PSBT, and relaxing the invariant that a PSBT represents a transaction (by allowing a PSBT to include no transaction data) would break many parsers and also just is antithetical to PSBT. It's not a general purpose format for sending data around - you could use protobuf if you wanted that.
If the goal is to have a machine parsable representation of descriptors, then by all means, do that. But it doesn't need to (nor do I think it should) interact with PSBT, other than having a different header that identifies it.
FWIW, the first draft was indeed PSBT with a different header: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-November/022184.html. https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2023-November/022186.html suggested we add descriptors to PSBT proper, which seemed like a good idea.
Would you support an alternative BIP that changed the header to make it intentionally different from PSBT files?
It's also unnecessary to have a descriptor in a PSBT. A PSBT can contain all of the data to derive it from the scriptPubKey at a later time. If an updater has the descriptor in order to put it in the PSBT, then it can also put the rest of the data in the PSBT so that the descriptor can be inferred.
How can the scriptPubKey infer a descriptor that contains, say, a key expression that ends in '/*'?
I just don't see how this proposal is at all useful.
The point of the BIP is to transfer descriptors between devices, in particular between a coordinator and (resource constrained) signing devices. It can be viewed as a codec for wallet policies.
Today, there exists a plethora of formats for exchanging descriptors, all with downsides:
- Raw xpubs are underspecified (missing script type)
- The BlueWallet textual format is inefficient (base58 xpubs) and doesn't support arbitrary descriptors.
- Raw BIP-380 descriptors are also inefficient and lack metadata.
- BCR-2023-010 are efficient but rely on the more complex (d)CBOR encoding.
- JSON encoded textual descriptors are also inefficient and require a JSON implementation.
This proposal addresses them all:
- Based on PSBT, signing devices need not spend additional code space on a separate codec.
- Efficient encoding of extended keys.
- Metadata (title, birth block) supported.
- Every current or future bip-380 descriptor can be represented.
Also, descriptors are a textual format, and encoding text in binary is just not very efficient. There are better ways to do that if you really wanted to.
We believe it's hopeless to keep a binary representation of output descriptors up to date with the moving target that is textual bip-380 descriptors (BCR-2020-010 attempted but failed). This BIP (along with wallet policies) address the worst offender by encoding extended keys efficiently.
Would you support an alternative BIP that changed the header to make it intentionally different from PSBT files?
That would be fine with me.
How can the
scriptPubKeyinfer a descriptor that contains, say, a key expression that ends in '/*'?
Although /* cannot be directly inferred, you can still make an educated guess. It only affects the ends of derivation paths from xpubs, so the last derivation index of a key expression with an xpub could just be replaced with /*. Although descriptors allow for xpubs without a /*, it seems unlikely to me that that would actually be used in practice.
But I don't understand why you'd need to have the parent descriptor at all. You'll have enough key origin information to derive any private keys and be able to sign.
This proposal aims to extend the capabilities of PSBTs by incorporating support for output script descriptors while also offering flexibility regarding the inclusion of the unsigned transaction within the PSBT structure. These enhancements could streamline certain Bitcoin transaction workflows and facilitate interoperability among different wallet implementations.
But I don't understand why you'd need to have the parent descriptor at all. You'll have enough key origin information to derive any private keys and be able to sign.
By example in the current ledger miniscript signing flow, the hardware device produce a Proof of Registration (PoR) against the parent descriptor that is stored by the softwarewallet (in order to keep stateless) the first time the descriptor is checked by user on signer side, then every time the software wallet send a psbt to sign to ledger device, it should also supply also the parent descriptor and PoR.
we previously discuss about this here
Hardware signing devices need knowledge of the full descriptor/wallet policy in order to:
- recognize change addresses
- recognize from which of your different accounts (sometimes called wallets) the inputs are spending from.
(The second could perhaps be considered a strengthening of the traditional guarantees of hardware signers, not sure if it was ever formalized before wallet policies; I think it's essential in practice for anything that is not single-user)
While it's true that PSBTs today have enough info to sign, they do not have enough info to do it securely - the descriptor is also needed. Therefore, I agree that it would be a great improvement to be able to pass this info via PSBTs; currently, the Ledger app's signing flow is sign(PSBT, wallet_policy), and quite some work in the app goes into figuring out change/address_index for each input and change output based on the derivations in the PSBT.
However, I think that's a separate discussion, and it's unclear that recycling PSBTs for a purpose other than representing transactions brings any benefit over than directly designing a space-efficient encoding specifically for descriptors or wallet policies – if the <30% space saving that can be obtained with a re-encoding is deemed important in practice.
Thanks for the prod, @jonatack . I've updated the proposal so it's no longer a PSBT extension but a separate format merely based on PSBT key-value maps.
I'd like to add proof-of-registration (PoR), but I'm not sure how it's best done. There's some discussion at https://github.com/wizardsardine/liana/issues/539#issuecomment-1877372827, where @bigspider suggests a HMAC scheme and in a later post @pythcoiner suggest that the format of PoR field should be vendor defined. I'm still of the position that to be maximally useful and interoperable, the PoR field should have a defined format and be a signature, not a HMAC.
I'd like to add proof-of-registration (PoR), but I'm not sure how it's best done. There's some discussion at wizardsardine/liana#539 (comment), where @bigspider suggests a HMAC scheme and in a later post @pythcoiner suggest that the format of PoR field should be vendor defined. I'm still of the position that to be maximally useful and interoperable, the PoR field should have a defined format and be a signature, not a HMAC.
As i already said in previous discussion, i do not have strong opinion about wether the PoR should be standardized or not, but i do not think it should be standardized in this BIP.
And anyway if i get it well both sheme (HMAC vs Signature) will be 32 bytes of data?
it should be interesting to have insight from other signing devices teams.
cc @benma @scgbckbone @stepansnigirev @stepansnigirev @JamieDriver
Also @kdmukai. Maybe others?
However, I think that's a separate discussion, and it's unclear that recycling PSBTs for a purpose other than representing transactions brings any benefit over than directly designing a space-efficient encoding specifically for descriptors or wallet policies – if the <30% space saving that can be obtained with a re-encoding is deemed important in practice.
I want to point out that while saving space is important to us, the ability to add metadata and eventually proof-of-registration is the more interesting feature in general.
I am in agreement with @bigspider's comment above.
BitBox02 libraries have functions like sign_psbt(psbt, wallet_policy) (example), and the policy/descriptor is required for anything but simple single-sigs.
It would be helpful if the way the policy is described and transferred to signers was standardized so apps don't have to implement different ways to interface with different signers.
PSBT seems like the obvious location to put this information. Another format that contains both the PSBT and the policy, which is interoperable with all software, seems difficult to achieve and impractical.
Given
- PSBTs are useful without descriptors, and
- descriptors are useful without PSBTs (backups, registration), and
- the combination of a PSBT and descriptor is useful for more secure signing flows,
Perhaps it's feasible to keep the formats separate, but concatenate them at the transport layer (QR code, files), in a backwards compatible way? Say, append the decriptor at the end of the PSBT, where devices without knowledge of descriptors may parse the PSBT and ignore the rest?
Thanks, Murch. I've added the mandatory headers, added a compatibility section, and expanded the rationale section.
It seems to me that the main source of contention is whether the format should be separate or part of PSBT. While convenient to extend PSBT, it's probably best to separate the format if for nothing else than descriptors are useful outside signing flows.
Proof of registration can wait for a future extension BIP, as suggested by @pythcoiner.
Thanks, Murch. I've added the mandatory headers, added a compatibility section, and expanded the rationale section.
It seems to me that the main source of contention is whether the format should be separate or part of PSBT. While convenient to extend PSBT, it's probably best to separate the format if for nothing else than descriptors are useful outside signing flows.
Proof of registration can wait for a future extension BIP, as suggested by @pythcoiner.
To be more accurate, i think how is processed the PoR should de defined in a separate BIP, but having an optional key-value pair defined in this BIP should be useful imho.
Thanks, Murch. I've added the mandatory headers, added a compatibility section, and expanded the rationale section. It seems to me that the main source of contention is whether the format should be separate or part of PSBT. While convenient to extend PSBT, it's probably best to separate the format if for nothing else than descriptors are useful outside signing flows. Proof of registration can wait for a future extension BIP, as suggested by @pythcoiner.
To be more accurate, i think
howis processed the PoR should de defined in a separate BIP, but having anoptionalkey-value pair defined in this BIP should be useful imho.
Good point. Done.
I'm very much PRO descriptors directly IN PSBTs. Even tho I understand @achow101 comments.
Regardless of the format here (which i haven't studied) it is useful to have descriptor (or at least policy) in PSBT. Example from Coldcard is when you want to have multiple miniscripts registered on device that have identical keys in miniscript but policy/script is different. Or same keys same policy but order of keys in policy is different. Here you just cannot infer solely from PSBT which wallet to use because keys match for all wallets...
why not just use PSBT_GLOBAL_PROPRIETARY key and send descriptor in there?
I'm very much PRO descriptors directly IN PSBTs. Even tho I understand @achow101 comments.
Descriptors in PSBTs are useful, but descriptors are also useful outside PSBTs: transferring descriptors between wallets, (long term) backups. It would be unfortunate to specify a descriptor format that only solved the signing use-cases.
why not just use
PSBT_GLOBAL_PROPRIETARYkey and send descriptor in there?
That would hijack PSBT_GLOBAL_PROPRIETARY, leaving it no longer available for vendor-specific uses.
Hey, wanted to check in to ask what the status with this proposal is. Was there progress on determining the direction it is being developed? Is it ready for an editor review? Please let us know if we can help.
We're (still) interested in pushing this proposal forward, given enough interest from one or more signing device vendors.
Sono di accordo Sono di accordo per eventuale sviluppo, credo che sia una buona descrizione.
I'm in favour of a separate format to PSBT, as is currently defined by this PR. There is nothing here that refers to a transaction, and wallet setup and registration is an entirely different process to transaction creation. Having a separate format simplifies how this format is detected and how it can evolve to serve future needs.
One thing to consider is whether it is worth expanding this format to also be used for collecting keys during setup, similar to how a PSBT is used to collect signatures during signing. This would allow a coordinator to specify at minimum a derivation path, and for signers to add keys to the format until it is complete. AFAIK there is currently no single format which allows for this, which makes geographically distributed multisig setup more difficult or vendor specific.
Another consideration is adding a BOD_GLOBAL_VERSION global type to specify the version, allowing for future iterations.
I also suggest adding an Encoding section which includes the file extension when serialized to disk, as in BIP174.
@seedhammer, I updated the PR title to that of the current "Title" field in this draft. Could you please verify if the PR description is current, and update it, if not?
@jonatack yes, the title is good, thank you.
Assuming Craig is willing to implement the completed format in his coordinator software, I believe the next step is to find a hardware signer vendor to OK the proposal (in principle) and work with them to implement it.
We've been too busy to lead the process, and as such the BIP is in limbo. Can it be left in draft, or should it be closed?
@seedhammer: Revisiting this, it is not required for a BIP to be implemented before it can be merged in Draft status. If you feel that your proposal is at a point where you have no further planned work and the next step is adoption, it seems that this PR should be ready for Editor review.
Whenever you are ready for editor review, please use the "Ready for review" button: