BOLT 1: Clarify optionality of message format `extension`

[tnull]

BOLT 1 currently states:

## Lightning Message Format

After decryption, all Lightning messages are of the form:

1. `type`: a 2-byte big-endian field indicating the type of message
2. `payload`: a variable-length payload that comprises the remainder of
   the message and that conforms to a format matching the `type`
3. `extension`: an optional [TLV stream](#type-length-value-format)

The `type` field indicates how to interpret the `payload` field.
The format for each individual type is defined by a specification in this repository.

as well as

The messages are grouped logically into five groups, ordered by the most significant bit that is set:
...
  - Custom (types `32768`-`65535`): experimental and application-specific messages

A few observations here:

• Stating that payload comprises the remainder of the message directly conflicts with there possibly being an extension in addition to said payload. If there is an extension, payload won't be the entire remainder.
• Stating that the type field indicates how the payload should be interpreted doesn't say anything about whether it also defines how extension should be interpreted.
• It's not entirely clear what 'an optional TLV stream' means. For who is it optional? Does the type define whether a reader should expect/support an optional extension, or is it expected that any application making use of custom messages is able to deal with such an extension being present (according to 'it's okay to be odd')?
• In contrast what's stated at the end of the first excerpt, (custom) message types are not necessarily defined in the BOLTs. At the very least there a those listed in the mentioned issue as well as over at https://github.com/lightning/blips/blob/master/blip-0002.md#messages

It would be great to get some insights into these details, and maybe even clarify them in the spec. In particular, I'd like to understand whether custom messages are expected to span the remainder of the message or whether they are expected to be, e.g., length-prefixed to always be able to handle an extension that could potentially be added by some other protocol.

For context, I'm raising these questions since the LSPS0 protocol as defined in bLIP-50 currently uses custom messages with a type|payload scheme, assuming that payload is always the remainder of the custom message and valid JSON. Is this in compliance with BOLT 1 or would we need to be able to handle the optional TLV extension?

[Roasbeef]

Stating that payload comprises the remainder of the message directly conflicts with there possibly being an extension in addition to said payload

Yeah I think the wording here is off.

This was added during a pass to migrate some of the older ad-hoc message extensions to be pure TLV.

payload is intended to be the base message type. With extension holding any additional TLV extensions. Some newer messages however are nearly entirely TLV messages, so in that case, the payload and extension fields are one and the same.

Stating that the type field indicates how the payload should be interpreted doesn't say anything about whether it also defines how extension should be interpreted.

The type field governs how both the payload and extension are to be interpreted.

It's not entirely clear what 'an optional TLV stream' means. For who is it optional? Does the type define whether a reader should expect/support an optional extension, or is it expected that any application making use of custom messages is able to deal with such an extension being present (according to 'it's okay to be odd')?

It depends on the parity of the TLV field itself. All messages can have extra TLV payloads appended to them. The reader is expected to retain and transmit that information, even if they don't understand it, as the extra payload may be covered by signatures (eg: for channel updates, etc).

IMO for uniformity, any custom messages added as part of bLIPs, should likely just be pure TLV messages.

[t-bast]

Indeed, this is poor wording in the BOLT because as roasbeef pointed out, TLVs were added after the "initial" spec and leveraged the fact that an optional extension was already defined to fill it with a per-message-type TLV stream.

The intent of this spec is to say that for each type, a specific payload is defined in the BOLTs whose size can be known from its definition, and if there are remaining bytes they must be a TLV stream extension that uses TLVs defined for this type.

If we had used TLVs from the very beginning, we wouldn't have included a payload at all and would have simply used TLVs everywhere, but we have to guarantee backwards-compatibility (and nobody wants to work on a painful migration of all message types just for the sake of cleaning up the spec...).

It's not entirely clear what 'an optional TLV stream' means. For who is it optional?

What this means to me is that the message is either just type || payload with the payload described in the BOLTs, or type || payload || tlv_stream. Whether we're in the first or second case is simply inferred by the message size: if there are remaining bytes after reading type and payload, then it MUST be parsed as a TLV stream. It then follows that it's ok to be odd for these TLVs, even for custom messages that aren't defined in the BOLTs.

It follows that if you add data after type and payload, this data MUST be structured as a TLV stream (with TLVs defined for this specific type).

any application making use of custom messages is able to deal with such an extension being present (according to 'it's okay to be odd')

Yes, in my opinion custom messages must also use this structure and handle it appropriately: if they find an even TLV type in the extension tlv_stream, they must consider the message invalid. But they also must ignore odd TLV types that they don't understand.

For context, I'm raising these questions since the LSPS0 protocol as defined in bLIP-50 currently uses custom messages with a type|payload scheme, assuming that payload is always the remainder of the custom message and valid JSON. Is this in compliance with BOLT 1 or would we need to be able to handle the optional TLV extension?

It would be more future-proof to support a tlv_stream extension, but it's your choice! The designers of LSPS0 can totally decide that it's unsupported to include an extension tlv_stream by never specifying any extension TLV for those custom messages.