[bug]: LND does not see transactions after the recovery procedure

[Perlover]

trafficstars

Background

My SSD drive died on my node. I had a fresh channel.backup file. But I haven't got to it yet, because LND after synchronisation doesn't see even old transactions inside the wallet. Although it is synchronised with the chain and network graph.

Your environment

• version of lnd 0.18.0-beta
• which operating system (uname -a on *Nix): Unix
• version of btcd, bitcoind, or other backend: bitcoind v27.1

Steps to reproduce

1. Create new .lnd empty directory with lnd.conf
2. Starting LND
3. I specified the password for the wallet, then specified the old seed and cipher seed phrase.
4. Waiting for LND to synchronise. But it shows walletbalance completely zero, and also does not see the first top-up transactions to the node's wallet (lncli listunspent).
5. Restarting LND does not help :(

Expected behaviour

I expect to see funds that were not in the channels (lncli walletbalance) as well as old transactions before recovery (lncli listunspent).

Actual behaviour

Once fully synchronised:

Once fully synchronised:

$ lncli listunspent
{
	"utxos": []
}

$ lncli listunspent
{
	"utxos": []
}

$ lncli getinfo|grep sync
    "synced_to_chain": true,
    "synced_to_graph": true,

I did the command lncli newaddress p2wkh and got a bitcoin address that shows my first transactions.

My public key is completely consistent with my old node. From this I conclude that the seed and cipher seed phrase are correct.

[Perlover]

I have now decided to try running lnd with the --reset-wallet-transactions option. I don't know if this will help yet. But in any case it all looks like some kind of bug.

[Perlover]

After rescanning the entire wallet again, there are some small changes. Namely:

1. lncli walletbalance - the balance is still zero. Although I'm sure there must be unspent UTXOs in the non-channels.
2. lncli listunspent - empty array
3. lncli listchaintxns - shows only two transactions. One is replenishment of the address I created for checking (I described it in the first post - lncli newaddress p2wkh), the second transaction is spending from that address.

By all my feelings, everything works like this: after the recovery procedure, LND for some reason did not generate a list of addresses from seed for checking and scanning balances on them and scanned incomprehensible addresses during recovery.

The second time when I created a new address (to see first message: "I did the command lncli newaddress p2wkh and got a bitcoin address that shows my first transactions."), which was replenished 2 years ago, and the subsequent scanning - LND checked and found transactions on it. But only at that address.

All other addresses still "do not exist" for it.

Accordingly, I can't even start the recovery procedure from SCB yet.

[Perlover]

I solved my problem only by downgrading the LND server to 0.17.5 (with a complete cleanup of the entire LND directory). The bitcoin backend remained the same - bitcoind v27.1.

As a result, the recovery of funds went smoothly and without problems.

I can conclude that version 0.18.0 has a bug that prevents the funds recovery procedure. Moreover, version 0.18.0 simply does not allow to restore funds from the wallet...

[ziggie1984]

Thank you for this detailed description of the problem, I will test the recovery procedure on regtest via 0.18 and get back to you.

[Perlover]

I noticed that Zeus 0.9.0-alpha11 has LND 0.18.0-beta inside it. If there is a bug in this LND version, this is very sad news including for Zeus users when they decide to restore their funds. I suggest to speed up the process of checking this issue. Ping @ZeusLN @kaloudis

[Perlover]

I decided to rename the issue title to a more proper meaning.

As a reminder, from my personal experience - version 0.18.0 and above, does not have a workable mechanism for restoring from static channel backup. I was able to restore funds in channels only after downgrading to v0.17.5.

So far I don't see anyone else checking this issue and confirming or denying this bug.

[guggero]

Can you provide some logs of a failed recovery attempt on 0.18.x? I don't think there's something wrong with recovery in general (that would've shown up in at least one unit and integration test) but maybe with the rescan on your end? Did you stop lnd during the initial recovery? That will abort the re-scan (an unfortunate but known issue) and won't restart again, so you'll end up with an incomplete state.

[Perlover]

Can you provide some logs of a failed recovery attempt on 0.18.x? I don't think there's something wrong with recovery in general (that would've shown up in at least one unit and integration test) but maybe with the rescan on your end? Did you stop lnd during the initial recovery? That will abort the re-scan (an unfortunate but known issue) and won't restart again, so you'll end up with an incomplete state.

I don't have the 0.18.0-beta logs at the moment. I deleted them when I completely deleted the ~/.lnd folder to start the restore again, but on version 0.17.5.

But I've scrutinized the logs and I can tell you that there was nothing unusual there. It looked as if I had restored a completely empty (without funds and transactions) LND server.

After, I executed the command to get a bitcoin address to deposit. And got the same address as I had gotten a long time ago. It was the same address that had been used to make the first deposits into the blockchain many years ago.

Then, I ran a full rescan of the wallet (after creating the address) and after that the LND noticed only two transactions related to the replenishment of the address I manually requested (i.e. it started tracking transactions on it only after the manual command, i.e. it served as a trigger). But the other transactions were not detected. I have a feeling that LND 0.18 tracks a different type of address (other BIP32 path) by default (maybe taproot?), and forgets the old transaction type (the LND's default type from 2022 year for my case) until you manually request it. Could this be the case?

Moreover, I decided to restore on another computer connected to a different host, and there also version 0.18 did not work. Then I deleted the ~/.lnd folder completely, put version 0.17.5 and the restore went smoothly the first time.

I'm pretty sure there is a bug in 0.18. Try in the tests to initially set the type of addresses that were in 2022 (on May - June months), the same version of LND that was in 2022, replenish the LND wallet. And then try to run recovery in version 0.18. You may not even open channels, because the problem didn't even reach SCB - it just didn't see the onchain wallet transaction.

[guggero]

Thanks for the additional info. I'll look into it a bit more then.

[ziggie1984]

Verified a small test setup with LND 18 and regtest, recovery of utxos worked as expected.

Worked with wallet inputs of:

• p2tr,
• p2wkh
• np2wkh

[ziggie1984]

@Perlover I think what would be the best, starting the recovery process again and sharing the logs which happened especially during the beginning of the rescanning process. (No matter if you already recovered the node, the addresses should still be scanned in the logs)

Should look similar like this but obviously with more blocks:

2024-07-15 21:58:41.296 [INF] LNWL: RECOVERY MODE ENABLED -- rescanning for used addresses with recovery_window=2500
2024-07-15 21:58:41.319 [INF] LNWL: Seed birthday surpassed, starting recovery of wallet from height=1 hash=4b4815da696b3134259ab6fdcfea32a45e20e96abba95af84c0cd18832f7f63c with recovery-window=2500
2024-07-15 21:58:41.926 [INF] LNWL: Scanning 308 blocks for recoverable addresses
2024-07-15 21:58:42.299 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:43.312 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:44.320 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:45.328 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:46.339 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:47.344 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:48.352 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:49.362 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:50.369 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:51.385 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:52.397 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:53.406 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:54.416 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:55.426 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:56.435 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:57.445 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:58.456 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:58:59.468 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:59:00.473 [DBG] LTND: Syncing to block timestamp: 2024-07-15 21:57:02 +0200 CEST, is synced=false
2024-07-15 21:59:01.204 [INF] LNWL: Recovered 1 external addrs at height=291 hash=05afc5e61c8a049983eeb5588edd6103dcae9035a5b8bffe82494f5cc4297a8c

Do you run a pruned bitcoind node ?

[Perlover]

I'm now running version 0.18.0 on that second machine where I was able to recover the node via version 0.17.5. I can now see in the logs that the addresses were successfully found. The process is still ongoing, but I can see in the logs that everything is now running successfully on LND 0.18.0 (lnd version 0.18.0-beta commit=v0.18.0-beta.1).

2024-07-16 01:15:58.653 [INF] LTND: We're not running within systemd or the service type is not 'notify'
2024-07-16 01:15:58.654 [INF] LTND: Waiting for chain backend to finish sync, start_height=852371
2024-07-16 01:15:59.658 [INF] LNWL: RECOVERY MODE ENABLED -- rescanning for used addresses with recovery_window=2500
2024-07-16 01:15:59.680 [INF] LNWL: Seed birthday surpassed, starting recovery of wallet from height=740517 hash=00000000000000000005335b0934e84d9b1b7e65752122d95501bea42cd9ff42>
2024-07-16 01:16:02.150 [INF] LNWL: Scanning 2000 blocks for recoverable addresses
2024-07-16 01:17:39.149 [INF] LNWL: Recovered 1 external addrs at height=740958 hash=00000000000000000006405ad43e0560eb23bed71d9aecfe99152856ed439d38
2024-07-16 01:17:39.150 [INF] LNWL: Recovered 1 internal addrs at height=740958 hash=00000000000000000006405ad43e0560eb23bed71d9aecfe99152856ed439d38
2024-07-16 01:17:39.150 [INF] LNWL: Found 2 spends from watched outpoints at height=740958 hash=00000000000000000006405ad43e0560eb23bed71d9aecfe99152856ed439d38
2024-07-16 01:18:07.671 [INF] LNWL: Recovered 3 external addrs at height=741159 hash=00000000000000000007d027a4c62da265413669fc168035be741327073805ad
2024-07-16 01:18:07.671 [INF] LNWL: Found 3 spends from watched outpoints at height=741159 hash=00000000000000000007d027a4c62da265413669fc168035be741327073805ad
2024-07-16 01:18:08.103 [INF] LNWL: Recovered 1 internal addrs at height=741161 hash=000000000000000000022009d26de3e15b0b1f6c62227938d5c284d62ade8621
2024-07-16 01:18:08.103 [INF] LNWL: Found 1 spends from watched outpoints at height=741161 hash=000000000000000000022009d26de3e15b0b1f6c62227938d5c284d62ade8621

Apparently there was an error somewhere on my side during the initial restores. Maybe in bitcoind communication or in its database. But at that time I tried using bitcoind on another machine and it did not restore. I suggest closing the ticket. If someone else has a similar problem, we can reopen it.

Do you run a pruned bitcoind node ?

In all cases described, bitcoind 27.1 was with a full blockchain base.

UPDATE: Maybe there was a problem after all. But, for example, the subsequent successful recovery of funds from all channels (closing them through SCB file) served as a trigger that the program code of LND is now working. This is, of course, my speculation. But what now distinguishes the node recovery procedure from the earlier one is that the node had unclosed channels then. And now all UTXOs are spent.

[ziggie1984]

Maybe there was a problem after all. But, for example, the subsequent successful recovery of funds from all channels (closing them through SCB file) served as a trigger that the program code of LND is now working. This is, of course, my speculation. But what now distinguishes the node recovery procedure from the earlier one is that the node had unclosed channels then. And now all UTXOs are spent.

Tested it with open channels now as well, recovery of the unspent utxos still works as required. If you encounter something similar again feel free to open another issue but I consider this now as resolved.