multi: update close logic to handle re-orgs of depth n-1, where n is num confs - add min conf floor

[Roasbeef]

In this PR, we address the oldest issue in the lnd track: #53. Incremental improvements have landed over the years, but this is the most significant one to date.

First, we start to scale confs for closes just like we do for funding confirmation. A small refactor helps us to re-use this, with additional tests added.

We revamp the chain watcher to implement a state machine to ensure that we recognize the most deeply confirmed transaction closure. Once we detect a spend, we'll register for N confirmations for that spend. If we detect anther one (we can get another spend if a re-org happens, and a new one confirms), then we'll re-register for confirmations for that spend. We also start to read from the NegativeConf channel which will be sent upon if the transaction is re-org'd out after confirmation.

It's important to note that the logic around the NegativeConf case is incomplete, as this will only trigger if: we wait for 6 confs, then we have a 7 block re-org. In that case, we've already claimed all of our outputs on chain typically. To handle this, we'll actually need to re-create the channel arb and the chain watcher, the re-handle any contested outputs we may have had. If the re-org goes all the way back to the funding transaction, then that's extremely dire, and needs special handling. This isn't covered in this PR.

Most of the diff is actually just tests: unit tests, property based tests (helps us ensure we can handle spend, conf, re-org, spend, conf of nearly arbitrary params -- eg: confirm coop, re-org, confirm force close, etc), integration tests, etc. The core change is in this commit: https://github.com/lightningnetwork/lnd/commit/b1d055edd48ff20c73ee9886ba7ac91be5065161.

This fixes most of https://github.com/lightningnetwork/lnd/issues/53. Re the second item, if the funding txn gets re-org out, then confirmed again, then we'll be able to detect that, and close it normally. We still need the active logic to detect such a case though.

[gemini-code-assist[bot]]

Summary of Changes

Hello @Roasbeef, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the robustness of channel closure handling within the lnd system, particularly against blockchain reorganizations. It refactors the chainWatcher to employ a state machine for managing channel closure spends, ensuring that transactions are only considered confirmed after a sufficient number of blocks, and gracefully handling reorgs by re-registering for spend notifications. Additionally, it introduces a configurable confirmation depth for testing and centralizes the logic for dynamically scaling confirmation requirements based on channel capacity, improving security for larger channels. A new development RPC is also added to facilitate more deterministic testing of UTXO sweeping.

Highlights

• Reorg-Resistant Channel Closure: The chainWatcher's closeObserver has been refactored to use a state machine for handling channel closure spends, making the process more robust against blockchain reorganizations by registering for confirmation and negative confirmation (reorg) notifications.
• Configurable Confirmation Depth: A new ChannelCloseConfs option has been introduced in ChainArbitratorConfig and chainWatcherConfig to allow overriding the number of confirmations required for channel closes, primarily for testing and development purposes.
• Scaled Confirmation Logic: The logic for determining confirmation requirements for channel closes (CloseConfsForCapacity) and funding transactions (FundingConfsForAmounts) now scales linearly with channel capacity, with a minimum of 3 confirmations for cooperative closes in production builds to enhance reorg protection.
• New Dev RPC for Sweeper: A TriggerSweeper RPC has been added to the devrpc service, enabling tests to explicitly trigger the UTXO sweeper to broadcast pending sweep transactions, which helps in managing asynchronous confirmation notifications in integration tests.
• Extensive Reorg Testing: Comprehensive property-based and scenario-based tests have been added for various channel closure types (cooperative, unilateral, breach) under different reorganization conditions, significantly improving test coverage for reorg handling.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

[^1]: Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

[lightninglabs-deploy]

@roasbeef, remember to re-request review from reviewers when ready