lightning
lightning copied to clipboard
ElementsProject
Recover funds from crashed core lightning node.
Background : I had an core lightning node where I closed all the channels and also closed the channel with node 02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1. For some reason it did not close the channel and my node crashed . I spun up a new core lightning node via Umbrel and copied my crashed node's hsm_secret and emergency.recover into the current node's lightning directory.
When I run emergencyrecover I get socket close. Also when I try to reconnect to node 02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1 I receive the below error Code 400: Unable to connect, no address known for peer
getinfo output
{
"id": "02ef10f80a4202c6766289345de16d462f365957c2d5df21edad6617e5830908cb",
"alias": "HAPPYCHASER-v23.08",
"color": "02ef10",
"num_peers": 1,
"num_pending_channels": 0,
"num_active_channels": 0,
"num_inactive_channels": 1,
"address": [
{
"type": "torv3",
"address": "dzzodtbpp2upywku6zzxvdqucjrwnvu7z76jparhwnwu2xtnuissysid.onion",
"port": 9735
}
],
"binding": [
{
"type": "ipv4",
"address": "10.21.21.96",
"port": 9735
}
],
"version": "v23.08",
"blockheight": 834107,
"network": "bitcoin",
"fees_collected_msat": 19292901,
"lightning-dir": "/root/.lightning/bitcoin",
"our_features": {
"init": "08a0800a0269a2",
"node": "88a0800a0269a2",
"channel": "",
"invoice": "02000002024100"
}
}
listpeers
"peers": [
{
"id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1",
"connected": false,
"num_channels": 1,
"features": "",
"channels": [
{
"state": "AWAITING_UNILATERAL",
"scratch_txid": "a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24",
"last_tx_fee_msat": 273000,
"feerate": {
"perkw": 376,
"perkb": 1504
},
"short_channel_id": "717548x1017x1",
"direction": 1,
"channel_id": "2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221",
"funding_txid": "20a2f38f5cc37c877a0ead2b52c4a90965a6fecb95ddef8d632bf6fdf4a1d62a",
"funding_outnum": 1,
"close_to_addr": "bc1q46nadmz4wr30n5vhzqxz6rhls83e5gd975lj9r",
"close_to": "0014aea7d6ec5570e2f9d197100c2d0eff81e39a21a5",
"private": false,
"opener": "local",
"closer": "local",
"features": [
"option_static_remotekey"
],
"funding": {
"local_funds_msat": 2516997000,
"remote_funds_msat": 0,
"pushed_msat": 0
},
"to_us_msat": 2491765413,
"min_to_us_msat": 1256508178,
"max_to_us_msat": 2516997000,
"total_msat": 2516997000,
"fee_base_msat": 109,
"fee_proportional_millionths": 1,
"dust_limit_msat": 546000,
"max_total_htlc_in_msat": 18446744073709551615,
"their_reserve_msat": 25169000,
"our_reserve_msat": 25169000,
"spendable_msat": 2465793413,
"receivable_msat": 62587,
"minimum_htlc_in_msat": 0,
"minimum_htlc_out_msat": 0,
"maximum_htlc_out_msat": 2491828000,
"their_to_self_delay": 144,
"our_to_self_delay": 144,
"max_accepted_htlcs": 30,
"state_changes": [
{
"timestamp": "2022-01-07T05:53:28.617Z",
"old_state": "CHANNELD_AWAITING_LOCKIN",
"new_state": "CHANNELD_NORMAL",
"cause": "user",
"message": "Lockin complete"
},
{
"timestamp": "2023-12-29T06:46:54.170Z",
"old_state": "CHANNELD_NORMAL",
"new_state": "CHANNELD_SHUTTING_DOWN",
"cause": "user",
"message": "User or plugin invoked close command"
},
{
"timestamp": "2023-12-29T06:46:54.953Z",
"old_state": "CHANNELD_SHUTTING_DOWN",
"new_state": "CLOSINGD_SIGEXCHANGE",
"cause": "user",
"message": "Start closingd"
},
{
"timestamp": "2023-12-29T07:00:16.241Z",
"old_state": "CLOSINGD_SIGEXCHANGE",
"new_state": "AWAITING_UNILATERAL",
"cause": "user",
"message": "Forcibly closed by `close` command timeout"
},
{
"timestamp": "2024-01-02T04:08:14.817Z",
"old_state": "CHANNELD_AWAITING_LOCKIN",
"new_state": "AWAITING_UNILATERAL",
"cause": "protocol",
"message": "channeld: received ERROR error channel 2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221: channeld: received ERROR error channel 2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221: Forcibly closed by `close` command timeout"
},
{
"timestamp": "2024-01-03T21:47:09.416Z",
"old_state": "CHANNELD_AWAITING_LOCKIN",
"new_state": "AWAITING_UNILATERAL",
"cause": "protocol",
"message": "channeld: received ERROR error channel 2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221: channeld: received ERROR error channel 2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221: Forcibly closed by `close` command timeout"
},
{
"timestamp": "2024-01-22T01:48:54.009Z",
"old_state": "CHANNELD_AWAITING_LOCKIN",
"new_state": "CHANNELD_SHUTTING_DOWN",
"cause": "user",
"message": "User or plugin invoked close command"
},
{
"timestamp": "2024-01-22T01:49:04.168Z",
"old_state": "CHANNELD_SHUTTING_DOWN",
"new_state": "AWAITING_UNILATERAL",
"cause": "user",
"message": "Forcibly closed by `close` command timeout"
}
],
"status": [
"Loaded from database"
],
"in_payments_offered": 758,
"in_offered_msat": 162519793342,
"in_payments_fulfilled": 15,
"in_fulfilled_msat": 1735415235,
"out_payments_offered": 1948,
"out_offered_msat": 394853748363,
"out_payments_fulfilled": 7,
"out_fulfilled_msat": 1760646822,
"htlcs": []
}
]
}
]
}
Can you share the logs when calling emergencyrecover? That might show us where it fails. Also adding @adi2011 as the author of the recovery plugin.
Please find the details below : I ran emergencyrecover and after that the getlog commands. Below is the info from getlog :
{ "created_at": "1710693901.278881343", "bytes_used": 3846881, "bytes_max": 10485760, "log": [ { "type": "SKIPPED", "num_skipped": 376 }, { "type": "INFO", "time": "13.808906673", "source": "lightningd", "log": "--------------------------------------------------" }, { "type": "SKIPPED", "num_skipped": 937 }, { "type": "UNUSUAL", "time": "19.311173752", "source": "plugin-bookkeeper", "log": "Snapshot balance does not equal ondisk reported 0msat, off by (+0msat/-0msat) (account wallet) Logging journal entry." }, { "type": "SKIPPED", "num_skipped": 9 } ] }
Additional Info for getlog debug :
{ "created_at": "1710694205.592952807", "bytes_used": 4241774, "bytes_max": 10485760, "log": [ { "type": "SKIPPED", "num_skipped": 161 }, { "type": "INFO", "time": "5.488185299", "source": "plugin-bcli", "log": "bitcoin-cli initialized and connected to bitcoind." }, { "type": "SKIPPED", "num_skipped": 120 }, { "type": "DEBUG", "time": "5.994206501", "source": "plugin-bookkeeper", "log": "Setting up database at sqlite3://accounts.sqlite3" }, { "type": "SKIPPED", "num_skipped": 576 }, { "type": "DEBUG", "time": "8.896661729", "source": "lightningd", "log": "Adding block 835094: 000000000000000000020a59df695df10af0f29757c7221c4b1faf5fe1510ecf" }, { "type": "SKIPPED", "num_skipped": 269 }, { "type": "DEBUG", "time": "10.607919907", "source": "plugin-bcli", "log": "Log pruned 134 entries (mem 10536455 -> 2427548)" }, { "type": "SKIPPED", "num_skipped": 43 }, { "type": "DEBUG", "time": "10.808193457", "source": "lightningd", "log": "Adding block 835099: 00000000000000000000182aa974d676e8e578eee4882a6f1c5dc2cc30e5005e" }, { "type": "SKIPPED", "num_skipped": 84 }, { "type": "DEBUG", "time": "11.336942878", "source": "plugin-bcli", "log": "Log pruned 135 entries (mem 10493227 -> 2033985)" }, { "type": "SKIPPED", "num_skipped": 34 }, { "type": "DEBUG", "time": "11.521482020", "source": "lightningd", "log": "Adding block 835101: 00000000000000000002d17bfbc3bbaa7a2e0ea1152d8bbe9ea6fec410ec82a1" }, { "type": "SKIPPED", "num_skipped": 2 }, { "type": "DEBUG", "time": "11.593491937", "node_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1", "source": "chan#4377", "log": "Got depth change 0->117554 for 20a2f38f5cc37c877a0ead2b52c4a90965a6fecb95ddef8d632bf6fdf4a1d62a" }, { "type": "DEBUG", "time": "11.593510787", "node_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1", "source": "chan#4377", "log": "Funding tx 20a2f38f5cc37c877a0ead2b52c4a90965a6fecb95ddef8d632bf6fdf4a1d62a depth 117554 of 3" }, { "type": "DEBUG", "time": "11.593568348", "node_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1", "source": "chan#4377", "log": "Funding tx 20a2f38f5cc37c877a0ead2b52c4a90965a6fecb95ddef8d632bf6fdf4a1d62a confirmed, but peer disconnected" }, { "type": "DEBUG", "time": "11.593580868", "node_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1", "source": "chan#4377", "log": "attempting update blockheight 2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221" }, { "type": "DEBUG", "time": "11.593582144", "node_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1", "source": "chan#4377", "log": "chain not synced, not updating blockheight" }, { "type": "DEBUG", "time": "11.593609273", "source": "lightningd", "log": "sendrawtransaction: 020000000001012ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a22001000000000c46af80028f62000000000000160014c02712a2d3f09e7efadfce3d4c76501ba53ed561650426000000000022002059cfd4bad3f6c8d6899b5bae169de4e3e3ff4a1e2a158a1bfd2f038e2ac50896040047304402206c3afe012479a046fe380785a64c0cccc2e928915a979ba9cb5453daa58acd8402200150871c063b95a625b9c887890d4308a617798f8bcba64e4896398780bf726c0147304402203bffba3835de33c37ae26d3b496e6625b54dbd39194c566b4b3138289604ef1602207f6469897e7971961d165280a2c4cd14bfde84849d5697b1aa207db96bcc8b7c014752210207c263af1717e8df4790d5dc3d433aefa98c0169286ad78ff46b17430d4bcdfa210239b38299018f11e10ec3deb11534da31d8cfc650b056fc70afc92299963a5fa452ae5835c120" }, { "type": "SKIPPED", "num_skipped": 1 }, { "type": "DEBUG", "time": "11.597354270", "source": "gossipd", "log": "REPLY WIRE_GOSSIPD_NEW_BLOCKHEIGHT_REPLY with 0 fds" }, { "type": "SKIPPED", "num_skipped": 1 }, { "type": "DEBUG", "time": "11.599591378", "source": "plugin-bcli", "log": "sendrawtx exit 0 (bitcoin-cli -rpcconnect=10.21.21.8 -rpcport=8332 -rpcuser=... -stdinrpcpass sendrawtransaction 020000000001012ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a22001000000000c46af80028f62000000000000160014c02712a2d3f09e7efadfce3d4c76501ba53ed561650426000000000022002059cfd4bad3f6c8d6899b5bae169de4e3e3ff4a1e2a158a1bfd2f038e2ac50896040047304402206c3afe012479a046fe380785a64c0cccc2e928915a979ba9cb5453daa58acd8402200150871c063b95a625b9c887890d4308a617798f8bcba64e4896398780bf726c0147304402203bffba3835de33c37ae26d3b496e6625b54dbd39194c566b4b3138289604ef1602207f6469897e7971961d165280a2c4cd14bfde84849d5697b1aa207db96bcc8b7c014752210207c263af1717e8df4790d5dc3d433aefa98c0169286ad78ff46b17430d4bcdfa210239b38299018f11e10ec3deb11534da31d8cfc650b056fc70afc92299963a5fa452ae5835c120) " }, { "type": "SKIPPED", "num_skipped": 1 }, { "type": "DEBUG", "time": "11.600410526", "source": "plugin-bookkeeper", "log": "account wallet has balance 0msat" }, { "type": "UNUSUAL", "time": "11.600419530", "source": "plugin-bookkeeper", "log": "Snapshot balance does not equal ondisk reported 0msat, off by (+0msat/-0msat) (account wallet) Logging journal entry." }, { "type": "DEBUG", "time": "11.600458255", "source": "plugin-bookkeeper", "log": "account 2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221 has balance 2491765413msat" }, { "type": "DEBUG", "time": "11.600461788", "source": "plugin-bookkeeper", "log": "Snapshot balances updated" }, { "type": "SKIPPED", "num_skipped": 15 } ] }
It seems your channel with 02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1 is already in AWAITING_UNILATERAL. Does it stub more channels in listpeers after running emergencyrecover?... @pratapc65
@adi2011 : I have closed all the channels and do not have other peer connections before the node recovery.. The channel with 2aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1 is in "AWAITING_UNILATERAL" for the past 2 months. The logs shows the raw transaction and how can I resend the transaction to mempool via bitcoin core .
"sendrawtransaction: 020000000001012ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a22001000000000c46af80028f62000000000000160014c02712a2d3f09e7efadfce3d4c76501ba53ed561650426000000000022002059cfd4bad3f6c8d6899b5bae169de4e3e3ff4a1e2a158a1bfd2f038e2ac50896040047304402206c3afe012479a046fe380785a64c0cccc2e928915a979ba9cb5453daa58acd8402200150871c063b95a625b9c887890d4308a617798f8bcba64e4896398780bf726c0147304402203bffba3835de33c37ae26d3b496e6625b54dbd39194c566b4b3138289604ef1602207f6469897e7971961d165280a2c4cd14bfde84849d5697b1aa207db96bcc8b7c014752210207c263af1717e8df4790d5dc3d433aefa98c0169286ad78ff46b17430d4bcdfa210239b38299018f11e10ec3deb11534da31d8cfc650b056fc70afc92299963a5fa452ae5835c120"
https://mempool.space/tx/a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24
Your transaction is already in the mempool, it hasn't closed for so long because the feerate seems very low.
@pratapc65 in the past when I had a forced closed chan with too low of a fee I had to use a "transaction accelerator" to pay miners out of band to include the transaction. There is viaBTC tx accelerator (they are expensive) and mempool.space has a tx accelerator service but I don't know if their service is active yet.
I have fell victim to stuck transactions before and even tried to replace unconfirmed transactions. Despite the popular myth, you cannot double spend transactions because nodes will reject the conflicting transaction. And transactions do not drop from the mempool. This claim is based off my experience with stuck transactions and experimentation from such scenarios. If I'm wrong, then someone should correct me on this. I think your best bet is to try to pay a miner out of band to include this transaction. viaBTC does accept a limited number of free entries for their accelerator service each hour. You could try with that first and if no luck then you can try the paid service.
Maybe this transaction can be RBF'ed or CPFP? But if itis a unilateral close that might not be an option.
Hello @nakoshi-satamoto : Thanks for the inputs. I tried to accelerate the transaction a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24 viaBTC free and it mentioned the fee was low. When I tried the paid service it mentioned the transaction was not recognized.
@cdecker ; @adi2011 : Is there a way to bump up the transaction fee via the btc core and rebroadcast .
@pratapc65 you can use withdraw RPC to initiate CPFP for this txn.
Please set the utxo argument appropriately.
@adi2011 ; Can you help me with the command. How do I know my nodes utxo that is stuck in mempool.
My nodes listfunds { "outputs": [], "channels": [ { "peer_id": "02aac548b877279c30f3abbb7301de93096e3b87144fc484dc3409bb0d6bd566b1", "connected": false, "state": "AWAITING_UNILATERAL", "channel_id": "2ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a221", "short_channel_id": "717548x1017x1", "our_amount_msat": 2491765413, "amount_msat": 2516997000, "funding_txid": "20a2f38f5cc37c877a0ead2b52c4a90965a6fecb95ddef8d632bf6fdf4a1d62a", "funding_output": 1 } ] }
You need to get a new address to transfer the amount, use lightning-cli newaddr
Then run: lightning-cli withdraw <newaddr> all normal 1 a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24:<vout>
Here vout is the output number (of your outpoint) in the transaction, 0 or 1 in your case...
@adi2011 : I get the below error when I run withdraw
withdraw bc1qfvjuwx9k5f2tsx50g**** all normal 1 a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24:1 { "code": -32602, "message": "Could not decode the outpoint array for utxos: "a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24:1" is not a valid outpoint array." }
Hi @pratapc65 , is likely that transaction a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24 is not in your node's mempool. Definitely is not in mine's
$ cln getutxout a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24 0
{
"amount": null,
"script": null
}
Trying to re-broadcast the transaction
$ bitcoin-cli sendrawtransaction 020000000001012ad6a1f4fdf62b638defdd95cbfea66509a9c4522bad0e7a877cc35c8ff3a22001000000000c46af80028f62000000000000160014c02712a2d3f09e7efadfce3d4c76501ba53ed561650426000000000022002059cfd4bad3f6c8d6899b5bae169de4e3e3ff4a1e2a158a1bfd2f038e2ac50896040047304402206c3afe012479a046fe380785a64c0cccc2e928915a979ba9cb5453daa58acd8402200150871c063b95a625b9c887890d4308a617798f8bcba64e4896398780bf726c0147304402203bffba3835de33c37ae26d3b496e6625b54dbd39194c566b4b3138289604ef1602207f6469897e7971961d165280a2c4cd14bfde84849d5697b1aa207db96bcc8b7c014752210207c263af1717e8df4790d5dc3d433aefa98c0169286ad78ff46b17430d4bcdfa210239b38299018f11e10ec3deb11534da31d8cfc650b056fc70afc92299963a5fa452ae5835c120
error code: -26
error message: mempool min fee not met, 273 < 524
I think package relay (https://bitcoinops.org/en/topics/package-relay/) is relevant to this conversation.
Is there a way I can change the status of the channel in channel DB and try to close the channel again .
@pratapc65 You need to put square brackets around the utxos "withdraw bc1qfvjuwx9k5f2tsx50g**** all normal 1 [a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24:1]"
@Lagrang3 minrelaytxfee can be used to reduce the minimum relay txn feerate in the mempool, right?
@Lagrang3
minrelaytxfeecan be used to reduce the minimum relay txn feerate in the mempool, right?
Nah, the problem was the mempool size that day. After 300MB (maxmempool) small feerates are evicted from the mempool. The threshold is computed dynamically, see glozow's explanation here. I've just tried getutxout and now and it worked,
today the mempool is not as full as it was 3 days ago.
$ lightning-cli getutxout a8c998d1ba872bc8003386ffda397c73acf60d74242af8ea0bce4dc8438cae24 0
{
"amount": "25231sat",
"script": "0014c02712a2d3f09e7efadfce3d4c76501ba53ed561"
}
For what is relevant to this issue, i guess the withdraw command would work as long as the bitcoin backend has the parent transaction in its own mempool.
But if the feerate is very low and the number of unconfirmed transactions being relayed in the network exceeds your local maxmempool (default is 300MB), as seen very often lately, then your tx could be out of your own mempool.
In that scenario even if you can locally increase your maxmempool to a higher value, if you broadcast a child transaction and your peer nodes do not have the parent, they will not accept the child because they will call it invalid. Package relay AFAIK tries to solve this problem. Just saying ... I am not sure what is the state of the art of this thing and I don't think this is included in core-lightning.
@pratapc65, you should try fee bumping again.
@pratapc65 closure confirmed in november 2024 (btw in the end your commitment was doublespend by your peer's commitment tx), you can come online and pick it up..
