[bug]: Self_Payment Failed

[lukegao209]

Background

Alice(Tapd0.5) Bob(Tapd0.5) Alice <----1 BTC Channel------>BOB Alice <----100000 USDT Channel------>BOB

Self_Payment for USDT Channel:
1. Generate 100 USDT Invoice On Alice 
2. Pay invoice on Alice 
3. Success

Self_Payment for BTC Channel:
1. Generate 0.00001 BTC Invoice On Alice 
2. Pay invoice on Alice 
3. Failed

Logs ： logs.txt

I encountered an issue with allow_self_payment. When Alice and Bob have both a BTC-Channel and a TaprootAssets-Channel, creating a BTC invoice on Alice and performing a self-payment on Alice fails. Could this be a bug with self-payment when both Taproot channels and BTC channels coexist? (If a Taproot invoice is generated, self-payment works fine.) Here are the logs for reference.

It’s particularly worth noting whether there is a way to avoid using Taproot channels for small BTC invoices. This is important because using Taproot channels for such payments might result in insufficient local BTC balance in the Taproot channel, making it impossible to pay Taproot invoices.

[Roasbeef]

When Alice and Bob have both a BTC-Channel and a TaprootAssets-Channel, creating a BTC invoice on Alice and performing a self-payment on Alice fails

When you go to make the payment, you can specify the outgoing and incoming scid's.

[Roasbeef]

What's the purpose of the self payment here? Is it to rebalance, or implement user multi-plexing (accounts) over channels? If the latter why does a higher layer keeping track of balances not suffice?

It’s particularly worth noting whether there is a way to avoid using Taproot channels for small BTC invoices.

You can specify the outgoing scid here to avoid using certain channels for normal BTC invoices.

[lukegao209]

What's the purpose of the self payment here? Is it to rebalance, or implement user multi-plexing (accounts) over channels? If the latter why does a higher layer keeping track of balances not suffice?

It’s particularly worth noting whether there is a way to avoid using Taproot channels for small BTC invoices.

You can specify the outgoing scid here to avoid using certain channels for normal BTC invoices.

我们的同一个节点的托管用户之间会产生相互支付invoice的情况，所以会产生selfpayment

[lukegao209]

@Roasbeef @guggero

Alice <---10000USDT--->Bob

When there is a USDT taproot channel between Alice and Bob, is it possible for Alice to transfer funds to Bob by paying an invoice? For example, if Bob issues an invoice for 100 USDT, can Alice pay that invoice using the channel, such that BTC price fluctuations are ignored and Alice can always use exactly 100 USDT to settle Bob’s invoice?

[guggero]

Yes, you can pay an invoice from your edge node directly with USDT. But the on-chain HTLC will always cost an additional 354 sats, which will go away once we implement https://github.com/lightninglabs/taproot-assets/issues/888. See https://github.com/lightninglabs/lightning-terminal/blob/a3ff9f23fde77769b654d4ac8c907fa4daec488b/itest/litd_custom_channels_test.go#L251.

[lukegao209]

Yes, you can pay an invoice from your edge node directly with USDT. But the on-chain HTLC will always cost an additional 354 sats, which will go away once we implement #888. See https://github.com/lightninglabs/lightning-terminal/blob/a3ff9f23fde77769b654d4ac8c907fa4daec488b/itest/litd_custom_channels_test.go#L251.

What I want to confirm is ： When Alice and Bob have a direct taproot channel, can Alice pay Bob’s USDT invoice without being affected by BTC price fluctuations?

[guggero]

Yes.

[guggero]

Did you get this working? Can we close the issue?

[lukegao209]

Did you get this working? Can we close the issue?

still doesnt work for me

[Roasbeef]

@ZZiigguurraatt IIRC you've made succesful self-payments with your test scripts?

[lukegao209]

Yes, you can pay an invoice from your edge node directly with USDT. But the on-chain HTLC will always cost an additional 354 sats, which will go away once we implement #888. See https://github.com/lightninglabs/lightning-terminal/blob/a3ff9f23fde77769b654d4ac8c907fa4daec488b/itest/litd_custom_channels_test.go#L251.

What I want to confirm is ： When Alice and Bob have a direct taproot channel, can Alice pay Bob’s USDT invoice without being affected by BTC price fluctuations?

For my tests: when Bob creates a 100 USDT invoice and Alice tries to pay that invoice using USDT, due to a 1% difference in BTC price, Alice is prompted that she needs 101 USDT. Since Alice and Bob have a direct channel, is there any way to avoid this issue so that Alice can still pay exactly 100 USDT? This is especially relevant when Bob generates a 100 USDT invoice using Alice as the edge node for price inquiries. @Roasbeef @guggero

[guggero]

What I want to confirm is ： When Alice and Bob have a direct taproot channel, can Alice pay Bob’s USDT invoice without being affected by BTC price fluctuations?

Hmm, okay, so my "Yes" above was a bit premature. What I meant is that when you have a direct asset channel and you send an asset payment, no conversion takes place. So the asset amount you send is exactly the amount that arrives. BUT what I forgot to mention is: Whenever you create an invoice, the asset amount is converted into sats (since it's a standard Lightning Network invoice where the only known asset is BTC). Then on the sender side the reverse calculation takes place. So there is a price fluctuation possible as you observe.

This is mostly because the use case for asset invoices is maximum compatibility across the whole network. The use case where sender and recipient have a direct channel is not the target use case for the invoice based payments.

What you can do, however, is a keysend payment. Example:

litcli ln sendpayment --dest <peer_node_ID> --keysend --asset_id <target_asset_id> --asset_amount 1

This will not include any price conversion from/to BTC and exactly 1 asset unit (plus the mandatory 354 satoshis needed for the HTLC itself) will be transferred. A keysend payment will insert an invoice on the receiver that then shows paid. So the order is reversed, the recipient doesn't create an invoice first.

So my question is: What's your exact use case? Do you require there to be an invoice from the recipient first? Or could you get your use case working with keysend payments?

[lukegao209]

What I want to confirm is ： When Alice and Bob have a direct taproot channel, can Alice pay Bob’s USDT invoice without being affected by BTC price fluctuations?

Hmm, okay, so my "Yes" above was a bit premature. What I meant is that when you have a direct asset channel and you send an asset payment, no conversion takes place. So the asset amount you send is exactly the amount that arrives. BUT what I forgot to mention is: Whenever you create an invoice, the asset amount is converted into sats (since it's a standard Lightning Network invoice where the only known asset is BTC). Then on the sender side the reverse calculation takes place. So there is a price fluctuation possible as you observe.

This is mostly because the use case for asset invoices is maximum compatibility across the whole network. The use case where sender and recipient have a direct channel is not the target use case for the invoice based payments.

What you can do, however, is a keysend payment. Example:

litcli ln sendpayment --dest <peer_node_ID> --keysend --asset_id <target_asset_id> --asset_amount 1

This will not include any price conversion from/to BTC and exactly 1 asset unit (plus the mandatory 354 satoshis needed for the HTLC itself) will be transferred. A keysend payment will insert an invoice on the receiver that then shows paid. So the order is reversed, the recipient doesn't create an invoice first.

So my question is: What's your exact use case? Do you require there to be an invoice from the recipient first? Or could you get your use case working with keysend payments?

One use case is: for example, when a user wants to deposit taproot USDT into Coinbase, the best experience would be for Coinbase to issue an invoice for 100 USDT and for the user to pay exactly 100 USDT. However, if the exchange rate keeps fluctuating, it could negatively impact the user experience.

If, when paying a taproot invoice, you allow specifying the rate via RPC, could that solve the issue?

[guggero]

The thing is, the invoice will never say "100 USDT", it will always show an amount in satoshis. I mean, sure, you could encode "100 USDT" in the metadata of the invoice and then react to that in the sender side application and try to send assets directly. But that will stop working as soon as there is at least one hop in the route.

If you use lnd's SendPaymentV2 method directly, you can encode the exact amount of assets directly in the FirstHopCustomRecords, even when using an invoice (and not keysend). But again, this will only work for direct channels. I'm not sure if you really want to add this extra handling for just the case of direct channels.

My take on the user experience impact:

• Any USD representing asset should be minted with a decimal_display of 6, meaning there will be fractional amounts possible. With that and a good/fair exchange rate calculation, the invoice might actually show as 100.01 USDT on the end user side. On top of that, they'll need to pay the Lightning Network routing fees (unless it's a direct channel again).
• The given exchange rate will be valid as long as the invoice is valid. So there's no fluctuation, just a small deviation from double conversion.

My next question would then be: How important is the "direct channel" use case to you? Are you planning on building an app around exactly the case where the recipient of the payments is always the direct channel peer? Or is this just an edge case you are trying to optimize for?

[ZZiigguurraatt]

@ZZiigguurraatt IIRC you've made succesful self-payments with your test scripts?

Yes, the changes required to enable this can be found here: https://github.com/lightninglabs/tapdvalidation/commit/da286e5c84c2be7a5b089a0ddd470c91e8e36d4e .

[ZZiigguurraatt]

@ZZiigguurraatt IIRC you've made succesful self-payments with your test scripts?

Yes, the changes required to enable this can be found here: lightninglabs/tapdvalidation@da286e5 .

I've done some more testing and I've found some problems with self payments which are described here: https://github.com/lightningnetwork/lnd/issues/9618 as they are LND specific and not taproot asset specific .

[bitcoin-coder-bob]

I'm actually witnessing different behavior than @lukegao209 where I create an asset invoice, and then often fail to self pay it with bitcoin. I outline my findings here: https://github.com/lightninglabs/taproot-assets/issues/1279

I have allow-circular-route enabled on my nodes, and am using AllowSelfPayment in the payment request. Note that in my case I do not want to route the payment along the same channel, but rather sats over one channel and asset over another channel (I acknowledge moving asset over a channel does mean moving 354 sats over the channel).

I believe the issue highlighted by @ZZiigguurraatt in the above comment is similar to the issue I have encountered. Where I find a difference is that the behavior I witness is flaky, in that the payment sometimes succeeds.

When I set Private: true on the invoice, the payment fails every-time with reason: FAILURE_REASON_INCORRECT_PAYMENT_DETAILS, and when I do not set Private I witness the flaky success behavior.

I tried setting 0 fees on the channels per this comment: https://github.com/lightningnetwork/lnd/issues/7417#issuecomment-1439341179 which was stated as a temporary solution for this issue: https://github.com/lightningnetwork/lnd/issues/7419

[guggero]

Thanks for the additional details.

When I set Private: true on the invoice, the payment fails every-time with reason

I think that is to be expected. Because private: true adds additional route hints which then interfere with the RFQ-specific route hint.

I do not want to route the payment along the same channel, but rather sats over one channel

Does that mean you are setting an outgoing channel ID that corresponds to a BTC-only channel? If yes, then that's good. If no, then that could explain the flakiness as perhaps sometimes the asset channel is chosen.

In any case, having the logs (on debuglevel=trace) for both sender and swap provider node (edge node) for the failure case would be useful to debug this.

[bitcoin-coder-bob]

@guggero

Does that mean you are setting an outgoing channel ID that corresponds to a BTC-only channel? If yes, then that's good. Yes. I log out the 3 channels I have (2 are sats, one is bobBux). My logs contain a mix of both nodes trace logs as well as some custom log lines I added.

Logs: https://gist.github.com/bitcoin-coder-bob/ac8411d98a330c37b973f6811ebc7960

These are their channel balances I log out before doing the payment.

checking local bal before (sats): scid: 533:1:0  chid: 586039697670144 cp: da087cf39f68aa5205f2f11fc2c35b16002d2f09ed05172f36202f253e19a12f:0   local bal: 9796530   remote bal: 200000
checking local bal before (sats): scid: 523:1:0  chid: 575044581392384 cp: f909be20fa8177e5a54d19e1a028bd25b5f002885e047d7524a1cd332c2840c7:0   local bal: 100000   remote bal: 9896530
checking local bal before (bobBux): scid: 513:1:0 chid: 564049465114624 cp: 3925e7b5bd1859a7964f435be6630bb62a89fa8c396e2c78bd60f2d4a04ae3cf:0  local bal: 0  remote bal: 96920    asset bal:0 private: true

These are their balances after the payment:

checking local bal after (sats):  chid: 586039697670144  local bal: 9776923   remote bal: 219606
checking local bal after (sats):  chid: 575044581392384  local bal: 100000   remote bal: 9896530
checking local bal after (bobBux): chid: 564049465114624   local bal: 0  remote bal: 96920     asset bal:0

The OutgoingChanIds field I set in SendPaymentRequest is an array that looks like:

[586039697670144 575044581392384]

You can see these 2 channels are the sat channels, indicating I do not have an asset channel as one of the OutgoingChanIds

Another log I added is for the hops taken. the pseduo code to get that looks like:

	for htlcIndex, htlc := range payment.Htlcs {
		route := htlc.Route
		for hopIndex, hop := range route.Hops {
			fmt.Printf("hop taken chid: %d pubkey:%s  bobBuxChan: %t  htlcIndex: %d  hopIndex: %d\n", hop.ChanId, hop.PubKey, hop.ChanId == bobBuxChid, htlcIndex, hopIndex)

and I get back:

hop taken chid: 586039697670144 pubkey:02c5fa27680f0d55d01ea51011e310735c5147473459d116f69d23242f01159c1d  bobBuxChan: false  htlcIndex: 0  hopIndex: 0
hop taken chid: 17716431482087996632 

We can see here the first hop taken is 586039697670144 which is a sat channel, and the second hop is the SCID of the channel with id 564049465114624 which id the asset channel. However no asset moved over the channel.

The key here is that the asset bal:0 is the asset channels CustomChannelData field's asset at index 0 local_balance field. Indicates the channel did not receive any asset from the payment.

[guggero]

I finally got some time to look into this today.

One question about the log: Is it possible this is the interleaved/combined version of both the sender and receiver litd nodes? If not, then something is very weird...

Second question: What version of litd did you try this on? I just created a circular re-balance (BTC to asset swap) integration test on latest master and it did go through every time, with the balance updating correctly. We did fix something in the invoice handling with the latest release, so perhaps that fixed this use case as well?

If you still see this behavior on master, then I think we need to take a closer look at your setup.

[bitcoin-coder-bob]

Apologies, I should have prefixed the node lines, the 2 node's logs were combined in the gist. New logs with prefixes for each of the 2 nodes (node1 and node2): https://gist.github.com/bitcoin-coder-bob/6e2558c0903498a1ca3b765f438fc868

This gist contains the logs from 5 self payments. Payments 1, 2, and 3 ran into the error, 4 and 5 did not (look for the line Swap result which will indicate the end of 1 of the payment lifecycles).

I am running a fork of lightning terminal, based off of v0.14.0-alpha, however I do not think the custom changes on my fork would have a material effect on the self-payment issue. I will need to do some rebase work to bump up to v0.14.1-alpha to see if that upgraded version resolves this issue.

[guggero]

Thanks for the clarification.

I think I have found an issue in tapd that might explain this. I'm still investigating where exactly this happens. But for now, I think if you remove the --lnd.allow-circular-route (i.e. make sure to NOT allow circular routes) flag, this should no longer fail.

[guggero]

Hmm, never mind... The removal of --lnd.allow-circular-route alone doesn't fix this. Investigating further.