ADR-002: Universal Share Encoding

[rootulp]

Description

ADR version of https://github.com/celestiaorg/celestia-app/issues/659

I plan on filling out the Implementation Details while working on the implementation but this is ready for review and I would definitely appreciate feedback

[rootulp]

Will update this ADR to replace the contiguous / non-contiguous share lingo in favor of reserved namespace / unreserved namespace shares because it is confusing when discussing contiguous shares in a message. Reserved namespace is also more precise as it matches the spec.

[musalbas]

I don't have any strong opinions on naming as long as it's clear. That being said, I would consider compact shares to have one "message", which further reinforces the idea that data in reserved namespace follow the same share-level format as any other namespace.

[rootulp]

What is the definition of "message"? I interpreted it as the previous name for data that a user submits to be included in a block (i.e. the second message in MsgPayForMsg which was renamed MsgPayForData). Maybe in this context, "message" means something else because we're now using it to describe evidence, intermediate state roots, and transactions.

the idea that data in reserved namespace follow the same share-level format as any other namespace.

Agreed and I like the share-level format terminology. I think we can split the format into:

1. Share-level
   1. namespace
   2. info reserved byte
   3. [if info reserved byte has message start indicator=1] message length varint
2. Application-level
   1. For sparse shares: no additional format imposed
   2. For compact shares: 1 reserved byte for location of first unit in the share and every unit is prefixed with a varint of the unit's length

if this separation adds clarity