Additional compatibility data

[fklebert]

In the current plain implementation zserio streams are only backward compatible if new content is added to the end of the stream. It is not possible to change structures in the middle of the stream without adding additional data. The additional data which we will call compatibility data for now, shall not be directly added to the stream but be an additional stream which older parsers may process. The original stream footprint shall not be touched since we want to still maintain a wire frame free format.

[mikir]

We will try to make the first implementation applying the following requirements:

1. Schema authors should not be forced to decide at the beginning which type is relevant for forward/backward compatibility.
2. Overhead (data and runtime) for clients which are fully compatible should be avoided.
3. Compatibility data should be the same for any older compatible version.

The following restrictions will be applied:

1. Field removal or type change is forbidden.
2. Only adding field/item at the end of structures, choices, unions, enumerations and bitmasks are allowed.
3. Enumerations which use whole range of underlying type cannot be extended.
4. Choices which have non-empty default case cannot be extended.

Our current solution will be storing the bit size of all compounds (encoded as varsize) in the independent data stream (called compatibility data).

In the first implementation, compatibility data will be compressed using RLE (efficient for array of compounds of the same bit size). Further size optimizations could be possible in the future on demand.

Note: Don't forget to implement handling of bit size zero in compatibility data. Bit size zero means that compound type is not extended at all (was and is empty).

Note: Don't forget to properly document this feature.