[epic]: Gossip 1.75 Implementation

[ellemouton]

This is the tracking issue for the Gossip 1.75 project (from now on called V2). The project aims at implementing the first feature bit detailed in the spec proposal. (replaces this. Replacement necessary due to the latest spec updates)

With the graph SQL-ization project now complete, the Gossip upgrade project can continue.

Background

Gossip in LND today(V1)

This section serves as a re-cap regarding how Gossip V1 works & how the persistence & logic layer of Gossip V1 works in LND today.

The Protocol Messages

These messages are used to announce & update information about nodes and channels in the graph. We store our representation of these messages in the lnwire package.

• Channel Announcements:
  • These serve to announce a channel on the network.
  • They use an SCID to indicate which output on the blockchain is the funding transaction of the channel in question.
  • They point to the 2 nodes on the network that own the channel.
  • Includes signatures to prove ownership of the UTXO in question.
  • These are never updated. There is only ever one channel announcement for a channel.
  • We monitor the blockchain to see when these utxos are spent, so that we know when we can forget about a channel.
• Node Announcements:
  • Once a node has proved ownership of at least one channel, they are allowed to send a node announcement which is used to broadcast more information about that node such as its addresses and the features that it supports.
  • Nodes are allowed to send new announcements in order to update previously advertised information.
  • In the V1 protocol, these messages are timestamped with a unix timestamp which cannot be too far in the future & must be newer than the previously received one.
• Channel Updates:
  • This message is used to communicate routing rules for a channel peer (fees etc).
  • It points to a channel (that we would know about via a previously received channel announcement) and uses a bit to indicate which channel peer this update was created by.
  • Nodes are required to keep a channel alive by occasionally sending a channel_update message for it.
  • These, like node announcements, are timestamped using a unix timestamp in the v1 protocol.
  • If we don't receive a channel update for a channel for long enough, we will mark it as a zombie in our DB (so that we don't keep trying to route through it).
• Announcement Signatures:
  • When we ourselves are creating a channel announcement, we need to negotiate the signature of the announcement with our channel peer. We use this message to do so.

The Flow of the messages & our internal representation

These gossip messages come through via our peers and we send them straight to the gossiper subsystem in their lnwire format. The gossiper does the various protocol level validation of the messages before converting them to the models formats. The models package in LND defines various types, namely: models.Node, models.ChannelEdgeInfo and models.ChannelEdgePolicy which is how we represent the various types in LND. These types also reflect more closely how the information is stored on disk.

The other source is from our local funding manager. It will send channel announcements & updates in the form of lnwire message to the gossiper even for un-announced channels.

The on-disk representation. Ie, the graph store

The following diagram shows the various tables we use to persist the graph info. At a high level, it is mainly just the node info, channels that point to those nodes and channel updates that point to channels and nodes. We also store some graph maintenance tables such as zombies, closed SCIDs and a prune log.
Some important points to take note of here:

• The main structures (nodes, channels and updates) all have a version field. At the moment, all persisted types will have version 1 to indicate that we received the v1 versions of these nodes/channels/updates. This field is in preparation for future gossip version.
• We have stored all the information required to rebuild the original lnwire messages so that we can send the valid gossip messages to any of our peers that ask us to sync gossip with them.

The use of the graph/gossip data in LND

We use this data in a variety of places:

1. The main reason for payment pathfinding.
2. To be able to perform gossip syncing with our peers who want to build the graph.
3. For various RPC endpoints (GetNetworkInfo, GetChannelInfo, GetNodeInfo).

Taproot Channels in LND today

Today, users can open Simple Taproot Channels in LND but are limited to keeping these as private channels.

The flow for opening such a channel is as follows:

1. Funding manager handles the initial open/accept channel flow to determine that the channel will be a simple taproot channel
2. As for any private channel in LND today, the funding manager then also constructs an unsigned lnwire.ChannelAnnouncement and a signed lnwire.ChannelUpdate and sends those to the gossiper.
   • The Channel Announcement uses the V1 message currently. It is unsigned because, since it is a private channel, we never actually broadcast this announcement. It is purely for our own persistence of the channel in the graph store.
   • The ChannelUpdate also uses the V1 message

Gossip V2

This section serves as a recap of what the new protocol entails.

Main purpose for the upgrade?

The main thing is so that we can advertise public taproot channels to the network & to take advantage of Schnorr signature aggregation. This cannot currently be done with the existing set of gossip v1 messages.

Other changes?

• The new messages use the pure-TLV format (with signed & unsigned ranges)
• The timestamps used for node announcements and channel update messages will now use block heights instead of unix timestamps.

Note on advertising P2WSH channels via V2

• The long term aim would be to get the entire network onto the new gossip protocol eventually.
• So the new messages can be used to announce the original P2WSH type channels. Both new ones and ones previously announced via the V1 protocol.

Interactions/overlap with the V1 protocol

• In terms of validating the messages, the protocols can be looked at completely disjointly. So for example: if we have channel A on protocol V1, that does not give the owning nodes the right to announce node announcements on the V2 protocol (and vice versa). This makes verification (ie, our gossiper code) fairly simple since really most of what we need to do is turn v1 message parameters into interfaces & then just make sure that we are addressing the correct persisted graph (v1/v2) when we do validation against our persisted data.
• Due to the above, it does mean that for both nodes & channels, we could have 2 records in the DB that have the same pubkey/channel ID but that use separate versions. This is important to remember.

Recap of the Feature Bit roll-out

In order to aid with the implementation of Gossip v2, the latest proposal includes a feature-bit rollout strategy. This lets us nicely break up the project into more manageable phases.

1. The first bit indicates an understanding of the new set of messages.
   1. Think of this as “full read capabilities” - the node is able to understand and validate channel announcements that advertise taproot channels & p2wsh channels.
   2. If this bit is announced along with the simple-tap-chan feature bit, then it also means that the node is able to open & announce public taproot channels with peers.
   3. This will also mean that the node is able to help peers to sync taproot gossip.
2. The second feature bit indicates that the node is able to re-announce existing public P2WSH channels using the new protocol.
3. Other feature bits:
   1. One to indicate to a channel peer that a node is perhaps willing/able to upgrade a previous private channel to a public channel. Since implementing the second feature bit above will involve quite a flow-change/refactor that makes it possible to re-init the signing of a channel_announceent long after funding time, this next feature bit should be easy to implement after that.
   2. A feature bit to signal that a node is willing & able to provide SPV proofs of a funding transaction along with the v2 channel_announcement message if asked.

High level project check list

The initial goal is to advertise feature bit 1. This means that the advertising node can understand the new gossip message and that if they advertise this along with the simple-tap-chans feature bit, then they can also advertise public taproot channels with the messages. We currently limit our scope focus to this goal only.

This goal can also be broken up into phases.

• [ ] Type Representation: Update our various type representations in preparation for the new messages.
• [ ] Validation & Write Logic: Update our validation logic (gossiper) to be prepared to validate the new messages (and write them to our DB)
• [ ] Read Logic: Various call-sites need to be prepped so that they are ready to read from a graph that contains messages across two gossip versions.
• [ ] Managing our 2 node announcements: We now have 2 Self nodes that we need to maintain & broadcast.
• [ ] Handling of Private Taproot channels: This includes dealing with any new private taproot channels & how to deal with existing private taproot channels.
• [ ] Opening and announcing Public Taproot Channels
• [ ] Updating Gossip Syncers to sync v2 gossip

Detailed check lists

1. Type Representation

• [x] update our lnwire V2 messages to correspond with latest spec
  - https://github.com/lightningnetwork/lnd/pull/10232 - https://github.com/lightningnetwork/lnd/pull/10267 - https://github.com/lightningnetwork/lnd/pull/10279

• [ ] update our DB schemas with the new columns needed for v2 - https://github.com/lightningnetwork/lnd/pull/10339

• [ ] update our internal type representation (models) to represent either version - https://github.com/lightningnetwork/lnd/pull/10339 - https://github.com/lightningnetwork/lnd/pull/10379 - https://github.com/lightningnetwork/lnd/pull/10380

• [ ] Implement conversions between all 3 types properly

2. Validation and Write logic

- [ ] Prep the "write" flow: ie, the gossiper should be able to verify all new gossip messages, validate them and persist.

3. Read Logic

- [ ] Prep various read-sites to handle the versioned types & merge them accordingly.

4. Managing 2 Node Announcements

- [ ] Node ann subsystem that manages updating and broadcasting 2 node announcements.

5. Private Taproot Channels

Today, users can use their LND nodes to open private simple taproot channels. These are stored under the v1 protocol currently and a feature bit is used to indicate that they are simple taproot channels. We want to have these channels instead be stored under the V2 version. The complexity comes in for existing private channels and dealing with channel peers that are not yet aware of the V2 protocol. More details below.

- [ ] Update priv-tap-chan creation flow to use new v2 messages - [ ] Correctly handle the case where our peer is still not aware of the new messages - [ ] write a migration to migrate priv tap chans on v1 to v2.

6. Public Tap Chans

- [ ] Define the new logic required for negotiating the musig sig of the v2 chan_anns

7. Gossip syncers

- [ ] update the gossip syncing messages with new v2 fields - [ ] update the gossip syncing logic to sync v2 gossip