[bug]: listunspent shows old and confirmed TX

[walmer-bolfet]

I am running a raspiblitz version 1.7.2

lnd version: 0.14.2-beta commit=lightning-terminal-v0.6.3-alpha

I have a discrepancy in my wallet balance, which is x sats - to which I can reconcile and believe is correct, and the sum of my listchaintxns and the sum of my listunspent (i.e., utxos), which are both x + y sats.

I was told then there was one utxo that equaled y sats, this was a sort of ghost utxo which might be remedied with a rescan of lnd, but alas, the rescan has increased the discrepancy and there are now three apparent ghost utxos that now sum to the y sats that are missing from the chain transactions. I looking for information on whether this is something to ignore or whether you have a fix to suggest. Thank you.

[walmer-bolfet]

Upon further investigation, I see these three ‘ghost uxtos’ are associated with balanced channels I opened with bos, as are the missing chain transactions which remedied, would reconcile the sums of uxtos and the sum of chain transactions to the balance of my wallet.

[Roasbeef]

So safe to close this issue?

[walmer-bolfet]

No, the issue remains even though I have provided some additional information about it.

[ziggie1984]

Could it be that those balanced channels never opened correctly. Bos open-balance works in a way that it first sends funds to another derivation path, which lnd does not take into account when showing wallet balance. Normally when you open the balanced channel you will see a refund transaction which you have to broadcast in case something goes wrong with the balanced channel opening. If you don't have it anymore, you can create a psbt which sends it back to your lnd default wallet account. Hit me up if you need help with it.

[walmer-bolfet]

All the channels have opened, and are routing and rebalancing as expected. Is there any other way to notice whether they opened incorrectly?

My reported wallet balance is correct. It is the sum of my utxos and the sum of my wallet transactions that reveal the issue.

[guggero]

Do you have any more information about those ghost UTXOs? Do the outpoints exist on chain (block explorer)? Do they exist in your lncli listchaintxns output? If they do, are they trying to sweep anchor outputs (small, 330 bytes outputs)? If yes, it might have been fixed by https://github.com/lightningnetwork/lnd/pull/6274 (which is in lnd v0.14.3-beta and will also be in the upcoming litd v0.6.6-alpha release).

[walmer-bolfet]

I have all the information about these uxtos that appears with the lncli listunspent command (i.e. address_type, address, amount_sat, pk_script, outpoint, confirmations). I don’t know how to use block explorer well enough to answer your question.

The transactions that I would have expected to see on my lncli listchaintxns output for opening these channels are not there, hence the discrepancy between my wallet balance and the sum of the chain transactions. However, there was a small chain transaction that concurrently appeared with the opening of these balanced channels, which I understood as part of the process. I don’t understand the meaning of ‘something is trying to sweep anchor outputs’.

[walmer-bolfet]

This is an example of what bos telegram reported with the channel opening: (I don’t understand how much can be shared securely, so I removed some information. The 141 sats fee paid does appear in my wallet transactions. There was another expected transaction for 1,050,095 sats that never appeared in the wallet transactions, but does appear as a ‘ghost utxo’. I believe I saw this transaction as pending for a short while while awaiting its confirmation.
This is similar to the other two balanced channel openings. —

⚖️ Received a 0.02100000 balanced channel open proposal from “New Channel” <Public Key> Proposed chain fee rate: 1/vbyte

⛓ (pending) Sent 0.00000141. Paid 0.00000141 fee. Sent to bc1qy63. . . . . , bc1q9ce. . . . .

⏳ Accepting new 0.02100000 channel from ’New Channel’ <Public Key> ‘My Node’

[ziggie1984]

Just checked my balance of all three and they add up:

lncli walletbalance | jq '.total_balance' = x lncli listchaintxns | jq '(.transactions[].amount|tonumber)' | awk '{total += $1} END {print total}' = x lncli listunspent | jq '.utxos[].amount_sat' | awk '{total += $1} END {print total}' = x

And I also checked my transactions I did with the bos open-balanced channel and everything is fine. So seems to me some transaction are either never been confirmed but have been created by lnd and therefore reside in the listchantxns command (see the bug guggero posted), to check this, just paste the tx_hash in mempool.space and see whether it finds a transaction for those ghost transactions. If it finds something then one should investigate further, bc then it could be that lnds wallet does not scan all derivation paths related to the rootkey.

[walmer-bolfet]

For me,

lncli walletbalance | jq '.total_balance' = x lncli listchaintxns | jq '(.transactions[].amount|tonumber)' | awk '{total += $1} END {print total}' = x + y lncli listunspent | jq '.utxos[].amount_sat' | awk '{total += $1} END {print total}' = x + y

Three 'ghost utxos' sum to y Three 'missing' wallet payment transaction sum to y

I have some information about the 'ghost utxos', as I said above. I have no information about missing wallet transactions so cannot proceed with your suggestion.

I cannot explain it, I can only see it, and it concerns me.

[guggero]

Can you send us some more information please? For example DM me on our public Slack, same username. It's really hard to say anything without any transaction IDs or on-chain information.

[walmer-bolfet]

Yes, guggero You should have it on Slack

[alexbosworth]

Did you try dropping utxos and then rescanning?

[walmer-bolfet]

I have not tried that. Clearly there is more information about why one would think these ghost utxos might be remedied by the dropping utxos than I understand. Insofar as I am able to add funds to my wallet and open new channels with my wallet without being effected by or effecting these utxos, I am awaiting better understanding before I try to fix something that appears to be working - but somehow displays this anomaly. Nevertheless, I thought those with better understand of lnd than I have would like to be alerted of this anomaly.

[xjmzx]

Seem to have a similar ghost, locked utxo, or wallet balance issue. The tx is confirmed (24000+ confs.) from a coop close 6 months ago. Shows up in listunspent and is visible in the mempool. Have tried two methods to create a transaction without success. Also walletbalance shows the discrepancy.

lncli walletbalance | jq '.total_balance' "(shows balance - the utxo)" lncli listchaintxns | jq '(.transactions[].amount|tonumber)' | awk '{total += $1} END {print total}' (shows balance including utxo) lncli listunspent | jq '.utxos[].amount_sat' | awk '{total += $1} END {print total}' (shows balance including utxo)

Seems similar, didn't want to duplicate. Any thoughts how to diagnose what is the utxo state / get balance back to normal? lnd v0.15-beta

[guggero]

@xjmzx Take a look at --reset-wallet-transactions.

Closing the issue as I believe the original problem was resolved. @rbpaul feel free to re-open if that isn't the case.

[walmer-bolfet]

I have no idea why you ‘believe’ this issue is resolved.

[xjmzx]

@guggero thanks for the suggestion.

Ran --reset-wallet-transactions which completed in a few hours. Oddly, it increased the number of utxos. All confirmed and spent in mempool. Otherwise, no issue. Just wanted to mention the result as I heard a few others mention the same issue.

[guggero]

I think the wallet will issue rescans for each found transaction output to find out if it was spent (which AFAIK is a background task independent of the initial wallet scan itself). So it might just take a while for them to be detected as spent. If they're still there in let's say a week, can you please comment back here?

[walmer-bolfet]

Mine ‘ghost’ uxtos have remained since I opened the issue in April.

[xjmzx]

@guggero my update on changes to balance / utxos.

The only change after a week or more is that another utxo was added in the list.

Originally there was just one. Then after reset-wallet-transactions seven more were found. After the last week (and db compaction) went up to nine in total.

So, just increasing number of utxo found as opposed to detecting any have been spent. They are all listed in mempool as spent. Presents no other problem. I'll keep an eye open and update in case any significant change.

[guggero]

Okay, thanks for the update. We'll need to take a look at the full logs to dig deeper.

Could you please set --debuglevel=debug and --reset-wallet-transactions on your node (again) and send us the full log since that last restart? You can DM me the logs on Slack or send me an email (in my profile). Also the full output of lncli listunspent and lncli listchaintxns would be very useful. Thanks!

[guggero]

Thank you very much @xjmzx for providing the full logs with the correct debuglevel. I think I was able to figure out what's going on: If one or more UTXOs are used for a channel open and no change output is created, the on-chain btcwallet library doesn't recognize the spend for some reason, because it doesn't get a change output to track (and the channel output is only tracked by the higher level lightning wallet).

So this is obviously a bug in the wallet that needs to be fixed.

@rbpaul can you confirm that your "ghost" UTXOs are also from channel opens where there was no change output? I know you sent them to me on Slack, but unfortunately the chat history there isn't kept. Could you please send me the output of lncli listunspent again so I can confirm this hypothesis?

[walmer-bolfet]

Greetings, The listupspent output of the 3 ghost utxos is below. As I previously mentioned, each one of them is associated with a balanced channel I opened with bos. Another point is none of the transactions show up when I look at my chain transactions, but my wallet balance show the correct amount. I will provide information as I am able with my limited understanding of what you are asking for. I don’t know how to determine whether ‘no change output’ was created. I will look forward to seeing a remedy.

utxos removed -

[guggero]

Thanks for the info. I can confirm they look the same as those from @xjmzx (two inputs, fully spent into a channel output with no change output). Now that we know what the actual problem is, we can start on a fix. So thanks again for the help in tracking this down.

[positiveblue]

@walmer-bolfet I will rename the issue to [bug]: listunspent shows old and confirmed TX now that we know what the actual problem is.

[yyforyongyu]

#6939 related?

[guggero]

I've been spending quite a bit of time trying to reproduce the problem. There seems to be a timing component to this so it only happens sometimes. It would really help a lot if I could take a look at the wallet file to find out how the data looks like in such a case. That way we could at least create a fix for the affected UTXOs (and that would maybe also give us a hint at how this happens in the first place). @walmer-bolfet or @xjmzx would either of you be willing to share the encrypted wallet.db file with me? Without the password I wouldn't be able to access funds, only see past transaction history. If yes, could you please DM me on Slack? That would be very helpful!

[walmer-bolfet]

@guggero

As I consider your request ( I am still learning), is this the file you request? ‘ /mnt/hdd/lnd/data/chain/bitcoin/mainnet/wallet.db’ -rw------- 1 bitcoin bitcoin 4194304 Nov 30 09:13 wallet.db

Please help me understand the difference between the wallet.db file and ‘the encrypted’ wallet.db file. Is a manual encryption of this file necessary? Why, if the password is not shared?

[guggero]

@walmer-bolfet thanks a lot for the offer, but I already got the wallet file from @xjmzx and found the problem (was able to reproduce :tada: ).

Please help me understand the difference between the wallet.db file and ‘the encrypted’ wallet.db file.

There is only one wallet file, the one you mentioned (wallet.db). Some parts of it are unencrypted while everything related to key material is encrypted within the file. So basically the transaction history and some info about the addresses used by the wallet can be examined but no private keys can be used to send funds without decrypting those parts with the wallet password.

For those interested, here's how these ghost UTXOs happen (this almost cost me my sanity, I didn't expect this to be so hard to pinpoint, but thanks to @xjmzx's file I was able to):

• A key from the internal lnd key scope (m/1017') is used to create a P2WKH address (which bos open-balanced-channel does to send temporary funds) and funds are sent to that address while creating a change output
• The funds are spent
• The user uses the --reset-wallet-transactions flag to rescan their wallet

So basically the problem is that bos open-balanced-channel sends funds to addresses of the internal scope that are then not spent by the wallet directly, which makes the wallet miss that spend TX. I'm working on a fix which should make it into lnd v0.16.0. Once the fix is in, any affected user will need to run --reset-wallet-transactions one more time to finally fix this.