bolts
bolts copied to clipboard
lightning
Add the ability to hold HTLCs before forwarding (FEAT 52/53)
This is an incredibly simple first start towards the protocol sketched out at [1]. It adds the ability to have a counteraprty hold an HTLC before forwarding it.
Specifically, the HTLC sender sets a required TLV on the
update_add_htlc message and sends an onion message to the final
recipient. From there, the final recipient uses the included
reply_path to notify the sender's counterparty that they're
online and ready to receive the HTLC.
In order to fully flesh out the protocol as sketched, we'll need to add an onion message mailbox service (which should be as simple as a new feature bit), add PTLCs, and extensions to BOLT 12 to allow signaling that the final recipient is often-offline. While we could add such signaling to BOLT 11, there's not a whole lot of reason to - if the recipient is able to provide an invoice, they're currently online!
I didn't bother basing this on the onion messages PR, but its a somewhat implied dependency.
[1] https://lists.linuxfoundation.org/pipermail/lightning-dev/2021-October/003307.html
Two more things that I think need more work:
- CLTV: using this scheme would require some very large CLTV deltas which is not desirable. For the channel Sender -> Hub A it's fine because the assumption here is that relay happens on this channel but for other channels in the route HTLCs should be short-lived. We should have a system for using CLTV deltas in the payment onion instead of an absolute CLTV, I have already suggested such a scheme for payments to blinded routes: https://github.com/lightning/bolts/pull/765/commits/9ca59ace8f14d6dd5996f5c934823dd0eca23d1d
- In addition to Hub A announcing support for this feature, Hub B must also support this. That should be advertised in the invoice.
If the recipient is online, this scheme adds an extra round trip for nothing. An alternative design would be try to relay the payment normally but add a reply path to Hub A in the payment onion, in the payload to Hub B. If the recipient is online, the payment goes through, if not then Hub B sends a hold_htlc to Hub A using the reply path and cancels the payment, the cancellation propagates back to Hub A and Hub A holds. Then when the recipient reconnects to Hub B, it sends a release_held_htlc to Hub A who retries the payment.
This eliminates an unnecessary round trip and falls back to a regular payment if Hub B does not support this scheme (we just need to put the reply path to Hub A in an odd TLV).
For this to work, we must have Sender -> Hub A -> ... -> Hub B -> Recipient where Hub A and B are always online. Hub A would hold the HTLC.
Yes, not sure how else it'd work.
You can't ask the recipient to tell Hub A when it gets online because for that you need the recipient to be online already to receive the instruction, it has to be Hub B that tells Hub A when recipient gets online. So when you create the payment, you need to know that the recipient is behind Hub B and that Hub B supports this hold protocol. And in the message to Hub B (with a reply path to Hub A), you need to say who is the recipient.
Yes you can, you just have to assume Hub B supports "onion message mailbox" support - ie store-and-forward. I noted this in the PR description/commit text.
CLTV: using this scheme would require some very large CLTV deltas which is not desirable.
Only on the first hop - given its only useful/used if Sender is often-offline, there's not really any reason to care about a high CLTV on the first hop - that capital wasn't going to be useful for anything anyway.
You're right that we'll need a further extension to allow Hub A to send the HTLC onward with a lower CLTV than the one in the onion, add that to the list of things we need to flesh this out fully :man_shrugging: .
Hub B must also support this.
See above, I don't think so.
If the recipient is online, this scheme adds an extra round trip for nothing.
Correct, that's part of why its super explicit that "this shouldn't be used if you think the recipient is online", that's also why its not specified for BOLT 11 - see above, it only makes sense with a BOLT 12 extension where you indicate that "Hub B can provide PTLC-based invoices on my behalf".
Yes you can, you just have to assume Hub B supports "onion message mailbox" support - ie store-and-forward. I noted this in the PR description/commit text.
You should specify this "onion message mailbox" in more details, I don't think it is as obvious and simple at you imply.
You're right that we'll need a further extension to allow Hub A to send the HTLC onward with a lower CLTV than the one in the onion, add that to the list of things we need to flesh this out fully man_shrugging .
That's what I meant and as I said I have already suggested a solution for that in the context of payments to blinded routes : use CLTV deltas in payment onions instead of absolute CLTVs.
Hub B must also support this.
See above, I don't think so.
Hub B needs to support "onion message mailbox", it won't work without it. So you need a way to advertise this support.
it only makes sense with a BOLT 12 extension where you indicate that "Hub B can provide PTLC-based invoices on my behalf".
The main (only?) use case I can see for private wallet to private wallet payment is sending money to someone you trust (family or friends typically). And this does not require BOLT 12 or PTLC, it can be done today with keysend.
The main (only?) use case I can see for private wallet to private wallet payment is sending money to someone you trust (family or friends typically).
With AMP, the payee can generate a static invoice and put it in an "always online" place (a QR in a website or can send an email or use any other asynchronous message mechanism). Without PTLC, the payer doesn't have any proof of payment, so she has to trust the payee. With PTLC no trust is needed.
You should specify this "onion message mailbox" in more details, I don't think it is as obvious and simple at you imply.
Not sure why this is complicated? Just store-and-forward. There's DoS questions, but not a lot we can do about that. In any case, yes, it should be specified, I'm just taking this one small step at a time.
That's what I meant and as I said I have already suggested a solution for that in the context of payments to blinded routes : use CLTV deltas in payment onions instead of absolute CLTVs.
Ah! Okay apologies, I misread. Yes, that option would address the issue as well.
Hub B needs to support "onion message mailbox", it won't work without it. So you need a way to advertise this support.
Yes, to be clear, this PR as-is is maybe 1/5th of what we need. It's not useful on its own, but it's a first step towards something great.
The main (only?) use case I can see for private wallet to private wallet payment is sending money to someone you trust (family or friends typically). And this does not require BOLT 12 or PTLC, it can be done today with keysend.
Huh? Ability to generate a reusable invoice and receive tips is a substantial use-case that people generally expect to work when they receive a QR code, and which doesn't work almost at all today. This is how bitcoin works and is often how its used. More generally, here is the US, this is how Cash App and Venmo non-bitcoin QR codes work, and people share those as well.
Further, ability to release HTLCs on receipt of a message could shift the mobile failure rate substantially - today a ton of the payment failures seen are just mobile nodes that happen to be offline. If the sender had an indication of when the recipient was online they could avoid a lot of the pain of terrible mobile-node-sending UX.
With AMP, the payee can generate a static invoice and put it in an "always online" place (a QR in a website or can send an email or use any other asynchronous message mechanism). Without PTLC, the payer doesn't have any proof of payment, so she has to trust the payee. With PTLC no trust is needed.
Ah, good point, we can just use keysend to avoid needing PTLCs. Of course we see still have to define a number of invoice extensions either way, and I'd generally prefer to do it on BOLT12 (a keysend option on BOLT 12 may annoy @rustyrussell but....) given we need onion messages to make this work anyway.
And this does not require BOLT 12 or PTLC, it can be done today with keysend.
Huh? Ah, good point, we can just use keysend to avoid needing PTLCs.
That's exactly what I wrote. I understand perfectly well why this hold HTLC scheme is a good feature for sending money to people you trust (which includes tipping because you trust the person you tip not to pretend that you didn't pay them). I'm just saying that there is no need to wait for PTLC, it can be done today with a new invoice extension, the scheme described here and keysend. And even though I like BOLT 12, I don't see how it makes things better in this scenario.
That's exactly what I wrote. I understand perfectly well why this hold HTLC scheme is a good feature for sending money to people you trust (which includes tipping because you trust the person you tip not to pretend that you didn't pay them)
Ah, okay, I totally misunderstood what you wrote there. Seems we're talking past each other a good bit here :(
I'm just saying that there is no need to wait for PTLC, it can be done today with a new invoice extension
Right, I think there are use-cases for proof-of-payment that make it useful to have PTLCs for this, though I agree its not required in most use-cases. Proof of donation is a valuable thing in some contexts, as is proof-of-payment as a scheme to improve security of offline/verifying signing in some cases.
And even though I like BOLT 12, I don't see how it makes things better in this scenario.
Right, I suggest it largely because it seems the obvious place to extend things. Given this work seeks to provide support for nodes which are behind an LSP, BOLT 12 is important for the same use cases - without BOLT 12, invoices for nodes behind LSPs are painfully large, with lots of readability issues. If this work didn't (a) already require new invoice extensions and (b) require onion messages anyway, I'd be considering BOLT 11 extensions or other invoice format extensions, but most of the work in implementing BOLT 12 is already required here (onion messages + blinded paths) and BOLT 12 is independently important for this use-case.
I believe the latest thinking for the name that people can refer to this idea is "async payments."
@TheBlueMatt @cdecker
It is worth exploring further how this proposal would work if it included support for the suggestion from @thomash-acinq to use keysend. Support for keysend would allow people to test much of the scheme before PTLCs become widely deployed by the network.
A few questions about using keysend:
-
would it be sufficient to use
offer_node_idof the Recipient from the BOLT 12 offer itself ? This would reduce the need for the Recipient's hub to issue BOLT 12 invoices for the offline Recipient. -
if the Recipient's hub DOES respond to an
invoice_requestonion message from the Sender with a BOLT 12 invoice for the Recipient, can the Sender still add a keysend TLV to their payment onion for the Recipient? even if the invoice uses route blinding? This would be a nice phase 2 that would give the Recipient privacy by not exposing their node ID. It would also be a step closer to a final PTLC based system.
An argument against keysend support as an intermediate solution is that it requires work that PTLCs will make obsolete, but if the effort is minimal and/or can be reused when PTLC support is available, it seems worth it.
Yea, we could consider it definitely. First we do need onion messages to be broadly available, which is still a ways off, then we'll need to define the onion message format in a second PR. Once the OM stuff lands as BOLTs I'll take a look at it (if I remember).
After some discussion with @arik-so, it seems like the current PTLCs scheme would not allow for reusable invoices where the receiver's LSP couldn't steal the money, unless PTLCs were adapted to have a keysend version, which then loses proof-of-payment (PoP).
The reason reusable PTLC invoices with PoP doesn't work is:
Given this is the scheme for PTLCs: https://imgur.com/28ssUuB where z is the secret that's sorta equivalent to today's payment preimages, you hit the same problem as HTLCs where once z is revealed, it can't be un-revealed, and trying to send a PTLC that is unlocked by the same secret would allow the receiver's LSP to steal funds. Hence why keysend PTLCs with a payer-provided z could "solve" this problem, but without PoP.
@TheBlueMatt thinks there should be a way to get PoP back with reusable-invoice PTLCs, though this scheme is as-yet unspecified. Either way, it seems best to just do HTLC-based async payments for now using keysend.
~Relatedly, note that HTLC keysend could get PoP by having the receiver sign the keysend preimage and send this signature back when fulfilling the payment.~ <-- see below comment, thanks @t-bast!
Looking forward to hearing people's thoughts on this. (@arik-so or other PTLC experts plz correct me if this is off)
@TheBlueMatt thinks there should be a way to get PoP back with reusable-invoice PTLCs, though this scheme is as-yet unspecified. Either way, it seems best to just do HTLC-based async payments for now using keysend.
My very high-level idea for this kind of feature was to use something similar to non-hardened BIP 32 derivation: if we had a way for the receiver to give a public key (+ some other crypto material, handwave handwave) to their LSP that lets the LSP independently derive public keys for which the receiver is able to derive the private key, it would be perfect. BIP 32 non-hardened derivation doesn't work, because as soon as the LSP learns the private key for one of the derived public keys, it can get all the future child private keys. I don't know if a scheme like that can exist or not, but it's probably worth doing more research to try to figure it out? It would be a nice challenge for researchers looking for something useful to do for lightning.
Relatedly, note that HTLC keysend could get PoP by having the receiver sign the keysend preimage and send this signature back when fulfilling the payment.
This isn't the same, because it relies on the receiver being honest: if they don't send a signature back, you've still paid (you can't reject the update_fulfill_ptlc) and you don't have any proof. It works when receivers are honest, so maybe in practice it's ok, but I hope we can do better.
My very high-level idea for this kind of feature was to use something similar to non-hardened BIP 32 derivation: if we had a way for the receiver to give a public key (+ some other crypto material, handwave handwave) to their LSP that lets the LSP independently derive public keys for which the receiver is able to derive the private key, it would be perfect. BIP 32 non-hardened derivation doesn't work, because as soon as the LSP learns the private key for one of the derived public keys, it can get all the future child private keys. I don't know if a scheme like that can exist or not, but it's probably worth doing more research to try to figure it out? It would be a nice challenge for researchers looking for something useful to do for lightning.
I think this type of scheme is fundamentally broken - the recipient's LSP cannot be involved in generating the payment point/hash or it can always double-spend. We could add an anti-double-spend blockchain, but, well....
The only idea I have for a direction that could work is for the sender to encode some unique info "payment sent by randomly generated pubkey X" into the payment point somehow such that the revealing of the payment secret is tied to X, but, like you, I have no idea if this is workable, I haven't done any real investigation here at all.
In any case, given the timeline for PTLCs, I don't think holding this up on PTLCs makes sense anymore. AFAIK no implementation has PTLCs on the roadmap, and only LND/LDK have taproot on the short-term roadmap at all. There are use-cases that make sense without PoP and we can switch to a PTLC-based system later.
And this idea? https://makers.bolt.fun/story/pitch-lightning-mobile-push-payments--489
And this idea? https://makers.bolt.fun/story/pitch-lightning-mobile-push-payments--489
That is solving a different problem - that talks about the recipient being online when constructing the invoice, something that we do not assume here. Also, that appears to describe roughly how some lightning nodes on mobile work today - they receive a notification from the LSP and hope that they get CPU time from the OS in response (not a guarantee, sadly) to handle the HTLC immediately by claiming it and showing the user a notification.
We could make a more general way for recipients to deliver data to intermediate hops, without relying on the sender to support the feature. The hop hint in the invoice could contain an encrypted TLV payload for the intermediate hop. This way recipients would be able to encode hold_htlc into the payload for the intermediate hop. The sender can include this data without having to know about this feature. Because this feature is really something between the intermediate hop and the recipient (it would have to know it needs to include the payload obviously).
If in the future there are other schemes that require the recipient to communicate something to an intermediate hop, this structure could be reused, and the sender wouldn't have to implement anything new.
That would still result in an HTLC getting sent a few hops to a blinded path introduction point and then holding liquidity on those hops into the recipient comes online.
That would still result in an HTLC getting sent a few hops to a blinded path introduction point and then holding liquidity on those hops into the recipient comes online.
It doesn't change this general flow at all (I think). I didn't mean blinded paths even. But it would provide a general way to let the recipient encode some data for an intermediate hop. So the recipient and the intermediate hop can make arbitrary agreements, without the sender having to know about it. The sender just puts the data in the payload for that hop. In this case, the recipient can encode that it wants the htlc to be held.
Another case (totally unrelated to this PR) could be a non-interactive lightning proxy. The recipient can encode the actual destination in the payload for the proxy.
There could be many different kinds of agreements between the recipient and the intermediate hop, that the sender doesn't need to know about.
Even in that case I don't think its really acceptable - senders really need to be aware that their capital is about to be locked up for potentially-hours before they hit send on the HTLC. An intermediate node should never do that to someone unless they're aware (or being malicious).
Even in that case I don't think its really acceptable - senders really need to be aware that their capital is about to be locked up for potentially-hours before they hit send on the HTLC. An intermediate node should never do that to someone unless they're aware (or being malicious).
Point taken. Agreed.