BOLT 7: querying for newer node announcements

[sstone]

There is currently no way to query for newer node announcements (*). This was not a problem  until recently since they almost never changed, and in practice were not used for computing routes

However, with new features such as Trampoline, AMP, .., path-finding algorithms will need to filter against features advertised in both channel announcements and node announcements, and node announcements can be expected to change more often (e.g. after updating to a new version).

To improve how to query and update node announcements, there are several options:

• add new specific query messages and filters, based on node ids
• extend existing queries to include optional node announcement timestamps and checksums (similar to what we have for channel updates)

I believe that the second option should be enough. It will allow nodes to quickly and efficiently update node announcements when channels from/to these nodes exist, and graph explorer will eventually get updated through the current rebroadcast scheme when nodes don't have any associated channels.

It may also be useful to add feature-based filter to the existing timestamp-based filter, so nodes will only receive channel and node announcements that match specific features.  

(*) you can ask your peer to include node announcement when you send them channel queries, but these are based on channel ids, not node ids, and timestamp filters are based on channel update timestamps, not node announcement timestamps.

[sstone]

As discussed, I'm adding more details about why this imho is an issue, and how to fix it:

Routing info is made of node announcements, channel announcements, and channel updates. The network can be modelled as a graph with node announcements as vertices and channel announcements + channel updates as edges. It also be used to choose which nodes to connect to.

• Node announcements include
  • static info: node id, ...
  • dynamic info: node features (and to some extend node alias and ip address)
• Channel announcements only include static info (channel id, node ids, ..)
• Channel updates only include dynamic information (fee and CLTV parameters)

As explained above, node features almost never changed and could be ignored in practice when computing routes. This is not true anymore yet we have no way to check and query for new node announcements. For nodes that are online most of the time it is not an issue, since they will eventually learn about newer node announcements though the regular gossip broadcast.

However, for nodes that are offline most of the time and remain online for just a few minutes at a time, which is the case for most mobile nodes, gossip broadcast is basically useless. This is why we proposed simple channel queries (until a better IBLT/minisketch... could be designed and implemented), which focus on channel announcements and udates and do not include ways to learn about and query updated node announcements.

=> I believe that there should be a way to query for "newer" node announcements

We could either create new query/reply messages that specifically target node announcements, and would be similar to what we have for channels, or modify existing channel-based queries.

Since node announcements cannot be broadcast if there are no matching channel announcements (i.e you don't announce nodes that don't have at least one open channel), channel ids can indirectly be used to identify nodes (as in short_channel_id:1 for node_id_1,short_channel_id:2 for node_id_2)

=> Adding new optionals TLVs and flags to existing query messages should be enough to let users learn and query node announcements

We could modify existing queries (query_channel_range and reply_channel_range) as follows:

• add a new Sender wants node timestamps to the existing query_option TLV in query_channel_range, at position 2
• if this option is set, receiver will include a new optional TLV that includes both timestamps for nodes 1 and 2

Since the same node can be referenced in different channel announcements, adding node timestamps for each short channel id would be very inefficient, and a smarter format the the list of timestamps should be used, for example a 1-byte bitfield, followed by timestamps for nodes 1 and/or 2 if the corresponding bit is set. With 10K nodes, and 30K channels, this would require an additional 10K * 8 + 30K * 1 = 110 Kb for the current network. The sender of reply_channel_range would have to do some work to avoid sending back duplicate info, but that should not be very expensive.

I'd like to collect feedback before I open a PR, the first question being: do we need to upgrade gossip/queries to better handle node announcements ?