rpc: got error `unable to find arbitrator` from `PendingChannels`

[yyforyongyu]

From the TODO, the error found in build:

=== RUN   TestLightningNetworkDaemonTemp/tranche03/06-of-99/bitcoind/data_loss_protection
    lnd.go:125: 
        	Error Trace:	lnd.go:125
        	            				harness_assertion.go:436
        	            				wait.go:48
        	            				wait.go:24
        	            				asm_amd64.s:1571
        	Error:      	Received unexpected error:
        	            	rpc error: code = Unknown desc = unable to find arbitrator
        	Messages:   	failed to PendingChannels for Carol

Here's what happened. In rpcServer, when we do fetchPendingForceCloseChannels,

1. We use r.server.chanStateDB.FetchClosedChannels fetches the pending force close channel.
2. After the channels are fetched, we then use r.arbitratorPopulateForceCloseResp which relies on the channel arbitrator to get the report.

Now, between steps 1 and 2, the arbitrator may be deleted due to the force close channel being resolved, which ends in this error unable to find arbitrator.

I think when receiving this error from chain arbitrator, it indicates the channel has finished the closing process and we need to remove it from the response.

Once it's fixed, the retry logic can be removed here.

[APBlitz]

I believe I might be running into this issue:

• lncli listchannels does not include the channel
• lncli closedchannes does not include the channel
• lncli pendingchannels returns [lncli] rpc error: code = Unknown desc = unable to find arbitrator and the logs show:

2022-11-01 22:14:59.181 [DBG] RPCS: [pendingchannels]
2022-11-01 22:14:59.245 [DBG] UTXN: NurseryReport: building nursery report for channel xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx:1
2022-11-01 22:14:59.245 [ERR] RPCS: [/lnrpc.Lightning/PendingChannels]: unable to find arbitrator

I was able to find the channel via chantools dumpchannels --closed with IsPending: (bool) true and see that it was included in block 761127 with CloseType: (string) (len=1) "1".

[APBlitz]

A re-sync seems to have fixed the problem.

[TrezorHannes]

A re-sync seems to have fixed the problem.

@APBlitz What type of re-sync did you do, could you be more specific? Like, rescan of the blockchain, with rescan=1, or rescan of the lnd wallet with lnd --reset-wallet-transactions?

Thanks

[APBlitz]

Hi @TrezorHannes apologies for not getting back to you earlier.

I believe I rescanned using --reset-wallet-transactions. It takes quite a while, but eventually all channels were back in their correct state.

[raelschiffler]

Hi @TrezorHannes apologies for not getting back to you earlier.

I believe I rescanned using --reset-wallet-transactions. It takes quite a while, but eventually all channels were back in their correct state.

Could you please inform the exact command you used to rescan?

[guggero]

Could you please inform the exact command you used to rescan?

https://github.com/lightningnetwork/lnd/blob/master/docs/recovery.md#forced-in-place-rescan

[SeverinAlexB]

We just experience this issue on our Blocktank regtest setup and --reset-wallet-transactions was not enough to resolve it. I had to stop lndmon and all services that call pendingchannels. Only then, --reset-wallet-transactions worked. Calling pendingchannels in the reset process would reliably not solve the issue. There must be some race condition.