electrum icon indicating copy to clipboard operation
electrum copied to clipboard

Cannot send lightning funds: Too many HTLCs already in channel

Open reivanen opened this issue 1 year ago • 29 comments

I decided to try lightning, and it automatically opened a channel for me. Everything seems correct, node is ONLINE, can send: shows correct amount. But every time i try to send i get this error:

Payment failed Too many HTLCs already in channel

electrum 4.5.2

reivanen avatar Jan 29 '24 22:01 reivanen

Is this with trampoline routing enabled or disabled?

SomberNight avatar Jan 30 '24 02:01 SomberNight

note: https://github.com/spesmilo/electrum/issues/8533 is conceptually similar

SomberNight avatar Jan 30 '24 02:01 SomberNight

No trampoline in use. But i remember trying clicking it, and sending few payments, but it said node does not support it, so i turned it off again.

I checked transaction history, and only transaction is the channel founding tx. So where are these HTLC coming from? Could the command line option enable_htlc_settle be of help here? (seems undocumented, as is the error code!?)

reivanen avatar Jan 30 '24 09:01 reivanen

lightning transactions that are 'in progress' are not displayed in your history. they only show up once the HTLC is settled.

edit: it would probably be good to display those in progress transactions in the history tab.

Could the command line option enable_htlc_settle be of help here? (seems undocumented, as is the error code!?)

no, that command is only useful for testing.

ecdsa avatar Jan 30 '24 10:01 ecdsa

When clicking on the node it shows only failed HTLCs. Is this what you refer to as transactions in progress? If so, can they still succeed? This would seem like extremely dangerous having transactions hidden that can still pay old invoices that are long gone. And if they cannot succeed, why keep them around, is it just to wait the CLTV timeout? If so, there certainly should be some indication of what is going on (besides a 5 second system popup with undocumented error code)

reivanen avatar Jan 30 '24 10:01 reivanen

if you have sent an HTLC, you cannot cancel it, you have to keep it around until it either succeeds or is failed. there is no danger of paying an old invoice twice, because the software won't attempt a new payment.

When clicking on the node it shows only failed HTLCs. Is this what you refer to as transactions in progress?

no, if they failed, they will not succeed. in progress HTLCs should be shown as "in current commitment transaction"

ecdsa avatar Jan 30 '24 10:01 ecdsa

since you mention that you are not using trampoline, can you explain what you mean by "it automatically opened a channel for me." ? Also, which node did you open a channel with?

It would also be nice to know what the value of max_accepted_htlcs of the remote node config. In the console tab, type the following: [c.config[1].max_accepted_htlcs for c in channels]

ecdsa avatar Jan 30 '24 10:01 ecdsa

It would also be nice to know what the value of max_accepted_htlcs of the remote node config. In the console tab, type the following: [c.config[1].max_accepted_htlcs for c in channels]

It would be useful to know both local and remote values (btw 1 is local). So please run this in the console instead:

[(c.config[1].max_accepted_htlcs, c.config[-1].max_accepted_htlcs) for c in channels]

SomberNight avatar Jan 30 '24 11:01 SomberNight

It would also be nice to know what the value of max_accepted_htlcs of the remote node config. In the console tab, type the following: [c.config[1].max_accepted_htlcs for c in channels]

It would be useful to know both local and remote values (btw 1 is local). So please run this in the console instead:

[(c.config[1].max_accepted_htlcs, c.config[-1].max_accepted_htlcs) for c in channels]

[(5, 483)]

With automatically opened a channel i meant that i just clicked in the gui and it did everything automatically, i did not specify node.

The "in current commitment transaction" is empty.

reivanen avatar Jan 30 '24 11:01 reivanen

[(5, 483)]

Hmm you must have opened that channel a long time ago, we increased the locally chosen value from 5 to 30 in version 4.1 (https://github.com/spesmilo/electrum/commit/762ebb12b253dd53517dbd99b13efcc0e0168b05).

SomberNight avatar Jan 30 '24 11:01 SomberNight

yes

reivanen avatar Jan 30 '24 11:01 reivanen

@reivanen you should close that channel, and open a new one with 4.5.2. It will use the increased value for that parameter.

ecdsa avatar Jan 30 '24 14:01 ecdsa

We should make the code smarter though, to realise that it should not split the payment into more parts if the channel cannot carry more HTLCs. And at the same time, probably also consider https://github.com/spesmilo/electrum/issues/8533

It is probably https://github.com/spesmilo/electrum/commit/35c9ac8f3173deaac14230ae1991a19d0d10987d that started causing the issue btw.

SomberNight avatar Jan 31 '24 04:01 SomberNight

is it a fix if i use electrum version before 4.5.2, or whatever included https://github.com/spesmilo/electrum/commit/35c9ac8f3173deaac14230ae1991a19d0d10987d ?

Sure, channel close&open new would work, but at current fee levels such unnecessary churn becomes pretty expensive.

reivanen avatar Feb 01 '24 13:02 reivanen

in general it is not possible to downgrade; older versions of the wallet use older formats for the wallet file, and cannot read newer versions.

if want to keep using this channel, you might have to wait until we have a proper fix, as @SomberNight suggested

ecdsa avatar Feb 02 '24 09:02 ecdsa

note: as temporary workaround, you can effectively disable the too eager splitting introduced in that commit by running wallet.lnworker.MPP_SPLIT_PART_FRACTION = 1 in the console (but this does not persist through application restarts)

SomberNight avatar Feb 02 '24 16:02 SomberNight

your suggestion resulted in electrum crashing while sending, and it offering me the privilege to send bug report. Which i did.

It took a long time to return the HTLC from in current commitment to available (failed) but eventually it did

edit: does the command expect spaces before and after the = ? This is how i copypasted it from here.

reivanen avatar Feb 07 '24 18:02 reivanen

your suggestion resulted in electrum crashing while sending, and it offering me the privilege to send bug report. Which i did.

I can't see a related new issue, it must have been merged into an existing one. Do you know have a link? Or if you have debug logs those would contain the traceback; or it is also possible to copy the traceback when sending the bug report (though at that point, you could just copy the link to the report).

edit: does the command expect spaces before and after the = ?

It does not matter.

SomberNight avatar Feb 08 '24 16:02 SomberNight

I did not open new issue thread since it was directly related to this thread.

this is what i found in log:

20240207T182433.564367Z |    ERROR | gui.qt.exception_window.Exception_Hook | exception caught by crash reporter
Traceback (most recent call last):
  File "/tmp/.mount_electrW4swOi/usr/lib/python3.10/site-packages/electrum/gui/qt/util.py", line 1426, in on_qt_callback_signal
    return func(self, *args[1:])
  File "/tmp/.mount_electrW4swOi/usr/lib/python3.10/site-packages/electrum/gui/qt/channel_details.py", line 156, in on_event_htlc_failed
    self.move('inflight', 'failed', payment_hash)
  File "/tmp/.mount_electrW4swOi/usr/lib/python3.10/site-packages/electrum/gui/qt/channel_details.py", line 126, in move
    row_idx = self.keyname_rows[fro].pop(payment_hash)
KeyError: b'\xc1\xb7\x0f\xfd9en%\xe4\xe3\x10C\xa5\x84\xc9\xd4\xda4>\xdaUT^\xcb&2B\xd8\xef\xe5k?'

but should this not already be in your knowledge as i sent the bug report electrum asked me to send?

reivanen avatar Feb 08 '24 16:02 reivanen

Actually digging through the log further, i saw this link mentioned by the crash reporter:; https://github.com/spesmilo/electrum/issues/5728

sad to see it is a 5 year old bug

reivanen avatar Feb 08 '24 17:02 reivanen

but should this not already be in your knowledge as i sent the bug report electrum asked me to send?

There are a lot of reports, which then get collated by a bot into issues. Sometimes the bot makes mistakes and merges reports into semantically different but similar-looking bugs. I can't know which report was sent by you.

Actually digging through the log further, i saw this link mentioned by the crash reporter:; https://github.com/spesmilo/electrum/issues/5728

Ah, yeah ok, that's a completely unrelated bug from this issue. That's just a refresh bug with the channel_details dialog in the gui. It does not explain why your payment failed.

SomberNight avatar Feb 08 '24 17:02 SomberNight

Is it safe to upload the whole log file somewhere or does it contain sensitive information?

reivanen avatar Feb 08 '24 17:02 reivanen

It is privacy-sensitive, but not security-sensitive. You can send email it to me if you want.

SomberNight avatar Feb 08 '24 17:02 SomberNight

I sent it.

reivanen avatar Feb 08 '24 17:02 reivanen

It is privacy-sensitive, but not security-sensitive. You can send email it to me if you want.

So, did you find anything interesting? (besides my privacy-sensitive information :)

reivanen avatar Feb 15 '24 21:02 reivanen

I had looked at your log but I could not really see anything interesting in it :/ You are trying to send 10 mBTC, which is quite a lot for a single HTLC. It might succeed if you retried several times. However for this kind of amount MPP really helps, and that is exactly at the heart of the issue here.

Setting wallet.lnworker.MPP_SPLIT_PART_FRACTION = 1 like I suggested, mostly disables MPP (hence single large HTLC). The code normally would try to split it into many parts, but this is not possible with your channel (as the old code set the parameters to too low when the channel was opened). So the best that can be done is what I said in https://github.com/spesmilo/electrum/issues/8857#issuecomment-1918354928, to make the code smarter to still split the HTLC, but only into as many parts as the channel allows.

SomberNight avatar Feb 22 '24 12:02 SomberNight

well, the problem does not seem to be the send amount, as 0.001 BTC results in same.

I will wait for next version and hope the fix is in by then, otherwise i will eat the loss and close the channel. (and truly hope it works unlike this first step into lightning using electrum). When i opened the channel it was just decent spending money, now it is more than a month's salary.

reivanen avatar Mar 30 '24 15:03 reivanen

If

wallet.lnworker.MPP_SPLIT_PART_FRACTION = 1

Does not solve the problem, is it expected for the next update to solve it? Seeing how ever more popular bitcoin is becoming day by day and how people pay ever higher fees, i'm thinking maybe eat the loss now and not wish on the next ethereum version fixing this?

Or is there someting more than the above mentioned command coming that could save this channel?

reivanen avatar May 06 '24 20:05 reivanen

with 4.5.5 the same error happened. As i expected because i did not see anything obviously related in the changelog. But after setting wallet.lnworker.MPP_SPLIT_PART_FRACTION = 1 i was able to send both 0.001 and 0.01 amounts. So either this is just random luck or there after all was some improvements made.

reivanen avatar May 30 '24 18:05 reivanen