[bug]: unable to retry coop close attempt with higher fee rate (between two nodes running v0.15.3)

[C-Otto]

Background

I tried to coop close a channel (lncli closechannel XXX 0 --sat_per_vbyte 1) which failed with [ERR] RPCS: [closechannel] unable to close ChannelPoint(xxx:0): unable to process close msg: latest fee proposal exceeds max fee: 0.00000602 BTC > 0.00000585 BTC. The channel appeared in lncli pendingchannels | jq .waiting_close_channels with an empty closing_tx.

Following attempts of coop closing the channel also failed:

unable to gracefully close channel while peer is offline (try force closing it instead): channel link not found

In my understanding, nodes running recent versions of lnd should be able to resume the close process.

The other node is running v0.15.3 with Thunderhub.

Note that I also have this issue with fixedfloat.com, although I don't know which version this node is running.

Your environment

• version of lnd: v0.15.3 with a few patches (Watchtower related & no more warnings as errors)
• Linux server 5.10.0-13-amd64 #1 SMP Debian 5.10.106-1 (2022-03-17) x86_64 GNU/Linux
• bitcoind v23

Steps to reproduce

1. attempt to coop close with fee suggestion too low for peer
2. attempt to coop close with higher fee

Expected behaviour

Channel closes, at least with higher fee.

Actual behaviour

unable to gracefully close channel while peer is offline (try force closing it instead): channel link not found

[C-Otto]

According to my logs, which only show the corresponding entries for the pubkey but not the channel point, my node tries to re-start the coop close using the same (low) fee rate. This makes me believe the channel will close eventually, once my peer accepts the low fee. However, I expect that my manual interventions (using a higher fee rate) influence these retries.

[C-Otto]

The RPC command fails here, before it can communicate the higher fee rate to any meaningful code in the lower layers:

rpcserver.go lines 2462ff:

                // If the link is not known by the switch, we cannot gracefully close
		// the channel.
		channelID := lnwire.NewChanIDFromOutPoint(chanPoint)
		if _, err := r.server.htlcSwitch.GetLink(channelID); err != nil {
                        rpcsLog.Debugf("Trying to non-force close offline channel with "+
				"chan_point=%v", chanPoint)
			return fmt.Errorf("unable to gracefully close channel while peer "+
				"is offline (try force closing it instead): %v", err)
		}

[C-Otto]

It seems the retries ignore any fee configuration provided via the CLI, but instead use the fee estimator (p.cfg.FeeEstimator.EstimateFeePerKW). In my case, this also returns 1 sat/vByte, but I think I'm able to fix this with a simple reconfiguration and restart.

[positiveblue]

Following attempts of coop closing the channel also failed

How long did you wait until trying to close the channel again? unable to gracefully close channel while peer is offline seems to indicate that the channel wasn't up. Is it possible that your peer closed the channel after you tried closing it for first time?

[C-Otto]

The "offline" part of the message is wrong, see code snippet above. The actual error is that "GetLink" doesn't return the link, which doesn't mean that the peer is offline.

I tried again a couple of hours later. It is not possible that the peer closed the channel as there didn't exist any close transaction on-chain (besides, as stated above, the channel was listed as "waiting_close_channels").

If you follow my comments in here, the actual bug seems to be that issuing lncli closechannel doesn't have any affect for channels that didn't close on first attempt. Furthermore, I was unable to close the channel by reconnecting to the peer because of wildly different fee estimates (I used 1000/ECONOMICAL, while my peer seems to use the defaults 6/CONSERVATIVE).

[C-Otto]

I was able to coop-close all pending channels by restarting lnd with 6/CONSERVATIVE (I tried 24/ECONOMICAL before, which didn't help). I don't think that paying 11.8 sat/vByte is necessary with an empty mempool (the previous 6 blocks weren't full!), but that's how life is until we re-organize the whole coop-close dance.

Please note that the original bug still exists: as a user I expect to re-try a coop close with fee settings specified at the time of requesting the interval, i.e. without resorting to "constants" defined in lnd's config.

[devastgh]

I was able to coop-close all pending channels by restarting lnd with 6/CONSERVATIVE (I tried 24/ECONOMICAL before, which didn't help). I don't think that paying 11.8 sat/vByte is necessary with an empty mempool (the previous 6 blocks weren't full!), but that's how life is until we re-organize the whole coop-close dance.

Please note that the original bug still exists: as a user I expect to re-try a coop close with fee settings specified at the time of requesting the interval, i.e. without resorting to "constants" defined in lnd's config.

As a cln user i totally agree with this. At this point i'm not going to coop close with lnd peers, if the delay is reasonable. I will just force close the channel with a 1sat/vbyte tx and wait a couple of days for the funds.

[BhaagBoseDK]

As a cln user i totally agree with this. At this point i'm not going to coop close with lnd peers, if the delay is reasonable. I will just force close the channel with a 1sat/vbyte tx and wait a couple of days for the funds.

force close fees would not be 1 sat/vb.

[Roasbeef]

It seems the retries ignore any fee configuration provided via the CLI, but instead use the fee estimator (p.cfg.FeeEstimator.EstimateFeePerKW)

That's correct, we don't currently persist the information from the past retry to disk.

[Roasbeef]

but that's how life is until we re-organize the whole coop-close dance.

We added the max fee setting for the initiator as people complained they had situations where they wanted to close with 1 sat/byte (as the initiator), but ended up accepting 20 sat/byte or w/e eventually due to our "satisficement algorithm". In this case, the initiator rejected things since it exceeded the max they wanted to pay.

Ultimately, the whole "closing fee negotiation" aspect of the spec is pretty ill defined. It's easy to end up in a state where neither side "compromises" so you end up in a deadlock. After we merge https://github.com/lightningnetwork/lnd/pull/7062, we'll likely go to a mode where the responder just accepts w/e fees the initiator wants, since they're paying fees at the end of the day, and the responder can always do CPFP assuming they have an output.

#7062 also starts to persist the delivery script that the RPC caller wanted to use, which can also be extended to commit their specified fee range.

[Roasbeef]

unable to gracefully close channel while peer is offline (try force closing it instead): channel link not found

The reason you see this is that when we go to co-op close, we remove the channel from the link so it can't forward any new HTLCs. We don't currently re-add it to the link when co-op close fails, as the assumption is the channel shouldn't be used any longer.

[VajraOfIndra]

I was able to coop-close all pending channels by restarting lnd with 6/CONSERVATIVE (I tried 24/ECONOMICAL before, which didn't help). I don't think that paying 11.8 sat/vByte is necessary with an empty mempool (the previous 6 blocks weren't full!), but that's how life is until we re-organize the whole coop-close dance.

Just had the same issue with 6/CONSERVATIVE rn. I closed 2 channels, got unable to process close msg: latest fee proposal exceeds max fee: 0.00000602 BTC > 0.00000585 BTC and after restrating LND, the channels closed at 15sat/vB (close to 3,000 sats of closing fees each) while the mempool is empty for a long time now.

[begetan]

I have the same issue here! I run lnd 0.15.4 backed with Neutrino backend.

I issued this command lncli closechannel <channel point 1 > --sat_per_vbyte 1 as I did many times before and got this error message:

[lncli] rpc error: code = Unknown desc = unable to process close msg: latest fee proposal exceeds max fee: 0.00000602 BTC > 0.00000585 BTC```

As a result all balance is in limbo and no way to to anything with it 

"waiting_close_channels": [
    {
        "channel": {
            "remote_node_pub": "026165850492521f4ac8abd9bd8088123446d126f648ca35e60f88177dc149ceb2",
            "channel_point": "xxx:1",
            "capacity": "8000000",
            "local_balance": "7914287",
            "remote_balance": "82695",
            "local_chan_reserve_sat": "80000",
            "remote_chan_reserve_sat": "80000",
            "initiator": "INITIATOR_LOCAL",
            "commitment_type": "ANCHORS",
            "num_forwarding_packages": "3",
            "chan_status_flags": "ChanStatusCoopBroadcasted|ChanStatusLocalCloseInitiator",
            "private": false
        },
        "limbo_balance": "7914287",
        "commitments": {
            "local_txid": "f2ca017b5d0ff1263f35edbfca29916dd13cdef2ca98f17161fba5b5da28fe6f",
            "remote_txid": "af47ec2e598e5ee79ec6dfe0de24a2abd045664fe58e684ea6baa8d50a9a3e32",
            "remote_pending_txid": "",
            "local_commit_fee_sat": "2357",
            "remote_commit_fee_sat": "2357",
            "remote_pending_commit_fee_sat": "0"
        },
        "closing_txid": ""
    }
]

[C-Otto]

You can fix this by using similar fee settings as your peer, and restart lnd. In lnd.conf set coop-close-target-confs=6 and bitcoind.estimatemode=CONSERVATIVE. Note that you might end up with more expensive coop-close transactions, which I think is better than having to force-close the channel.

In my understanding a coop-close attempt works if the two (lnd) peers start suggesting values where the difference between the two values is at most 9 (each peer is allowing 1/3x to 3x). You can call bitcoin-cli estimatesmartfee 6 CONSERVATIVE etc. to see the fee rates that are used in this computation.

[begetan]

@Roasbeef

The reason you see this is that when we go to co-op close, we remove the channel from the link so it can't forward any new HTLCs. We don't currently re-add it to the link when co-op close fails, as the assumption is the channel shouldn't be used any longer.

We don't want to use this channel anymore, right but since cooperation close has failed due to fee estimation error it should stay alive. We can change fee rate and re-try it again, or keep working.

I do not understand how the channel balance could be in Limbo without any transaction issued into the Mempool?

How to recover this funds from a Limbo state?

[C-Otto]

How to recover this funds from a Limbo state?

See my previous comment.

As far as I understand the specification, reviving a channel isn't possible. The transaction isn't in the mempool because it doesn't exist. You can force-close the channel and send the previously signed transaction into the mempool, or you can reconnect to the peer and try creating a coop-close transaction again.

[begetan]

I am trying to use your solution. After restart there is an error message in the log: unable to process close msg: latest fee proposal exceeds max fee: 0.00000644 BTC > 0.00000624

@C-Otto

or you can reconnect to the peer and try creating a coop-close transaction again

What do you mean? My lnd is already connected to the peer and I am not able to to close it again: [lncli] rpc error: code = Unknown desc = unable to gracefully close channel while peer is offline (try force closing it instead): channel link not found

Have you closed cooperatively your transaction from this topic finally? How long did you wait?

[begetan]

It sounds crazy, but I've managed to coop-close channel by restarting lnd several times.

One important thing in my case might related to Neutrino backend configuration. Although I switched over the neutrino backend to avoid it's influence, but I missed a top level configuration of feeurl setting. It was configured as feeurl=https://nodes.lightning.computer/fees/v1/btc-fee-estimates.json and probably overrode bitcoin core method. Thus after several restarts, lnd was able to get a compatible fee.

@Roasbeef this user experience should be improved somehow

[carlaKC]

We've been spelunking around with signet nodes as part of our FOSS program and ran into this error on 0.17.3-beta.

Error:

lncli closechannel --chan_point "99b448ef80ac4f3b7fbb44f15d768f0b24e61a5815c24f66f443104ca1ba0440:0"
[lncli] rpc error: code = Unknown desc = unable to process close msg: latest fee proposal exceeds max fee: 0.00000468 BTC > 0.00000453 BTC

Deets for the channel we tried to close:

2024-02-07 18:56:08.588 [DBG] HSWC: Requesting local channel close: peer=0324b10f41493935f35cd3b4c1131c94e8f83bfe66c6ebf842790e59e182076d30@35.209.148.157:9735, chan_id=4004baa14c1043f4664fc215581ae6240b8f765df144bb7f3b4fac80ef48b499

I have logs for both sides of the channel, this stands out to me after receiving ChannelReestablish:

2024-02-07 21:29:26.084 [ERR] PEER: Peer(037e148d23601b776c967af011e17328e8dce96e5dde43a29cc896555c5c75fa54): resend failed: unable to fetch channel sync messages for peer 037e148d23601b776c967af011e17328e8dce96e5dde43a29cc896555c5c75fa54@xxx: unable to find closed channel summary

Logs have some IP addresses in them so don't want to upload here, but I'll DM them to whoever on slack - just let me know!

[Roasbeef]

@carlaKC

https://github.com/lightningnetwork/lnd/blob/935e550da67b4df17d29c830c5ea66a02c84af3f/lnwallet/chancloser/chancloser.go#L768-L774

https://github.com/lightningnetwork/lnd/blob/935e550da67b4df17d29c830c5ea66a02c84af3f/lnwallet/chancloser/chancloser.go#L309-L317

This is a protocol issue IMO. It'll be fixed by: https://github.com/lightningnetwork/lnd/pull/8453

What's happening is that the initiator, who pays fees, is out right rejecting an offer from someone else, as they want to pay close to 1 sat/byte, and proposing at 5 sat/byte is 5x that. With rbf coop close, either side can always get a sign close at the fee rate they desire if they have the funds to pay for it.

[carlaKC]

What's happening is that the initiator, who pays fees, is out right rejecting an offer from someone else, as they want to pay close to 1 sat/byte, and proposing at 5 sat/byte is 5x that.

Interesting! Weird that we ran into this on a signet (with standard lnd/bitcoind confgis), would have though that fee estimation would be the same but I know there are some inconsistencies in core fee estimation so perhaps we hit that!