M17-Project
Reorganize and define `TYPE` bit fields
- Reorganize
TYPEbit fields such that the Channel Access Number is byte-aligned - Add
METApayload type definitions for the Packet Mode
I thought I would float out this idea for restructuring the LSF's Type Field. Forgive the screenshot, but its the best way I could keep the "formatting" of it intact, but this is an idea I was mulling over, its not set in stone, but if a change must be made, I would try to capitalize on it by doing something like this:
For context, this is the Discord discussion that led to this issue being created: https://discord.com/channels/771492414120656907/877593757799813131/1408043688059207680 In particular, @silseva 's idea here is relevant: https://discord.com/channels/771492414120656907/877593757799813131/1408067730396942347
The proposed changes are:
- merge the data type and stream/packet to a unique 3-bit field (because packet is a data type)
- include signed stream mode into the encryption type field, extending it to three bit
- align CAN to the boundary of byte 0, with the four bits 0...3
The rationale is that aligning the CAN simplifies handling of TYPE data, both in the case you use a struct or bit manipulation. Structs are always preferrable to bit manipulation (for readability, ...) and the fields can be organized in the definition to have the correct endianness of the output data.
The proposed changes are:
* merge the data type and stream/packet to a unique 3-bit field (because packet _is_ a data type) * include signed stream mode into the encryption type field, extending it to three bit * align CAN to the boundary of byte 0, with the four bits 0...3The rationale is that aligning the CAN simplifies handling of TYPE data, both in the case you use a struct or bit manipulation. Structs are always preferrable to bit manipulation (for readability, ...) and the fields can be organized in the definition to have the correct endianness of the output data.
The proposal I made above covers all of those bases, I believe, and gives us a bit of future proofing for new META types and call types, and so on. Unless there is something in it that isn't to your liking, I am willing to make adjustments. My proposed opcodes aren't set in stone, and as-is, would need one adjustment, would need to include a NULL META opcode and give it value 0x00, which my "napkin sketch" there had 0x00 for Text Data.
I believe my proposal also helps to satisfy this suggestion as well, since we could allocate a new META opcode for this as well.
I'm okay with the proposal of @lwvmobile, but how about "Payload Opcodes" instead of "Call Opcodes", moving them to the most significant side of the byte, keeping the existing Data Type values (0x0..0x3), especially the reserved 0x0 value. That way any V3 TYPE will be immediately distinguishable from a legacy TYPE. These suggestions will help when I create a new client type in mrefd to handle hot-spots/repeaters using the new TYPE format. (HINT: V3 clients will connect to mrefd with CON3 instead of CONN, mrefd will transcribe TYPE data so old clients can still talk&listen&message with new clients.)
I'm okay with the proposal of @lwvmobile, but how about "Payload Opcodes" instead of "Call Opcodes", moving them to the most significant side of the byte, keeping the existing Data Type values (0x0..0x3), especially the reserved 0x0 value. That way any V3 TYPE will be immediately distinguishable from a legacy TYPE. These suggestions will help when I create a new client type in mrefd to handle hot-spots/repeaters using the new TYPE format. (HINT: V3 clients will connect to mrefd with
CON3instead ofCONN, mrefd will transcribe TYPE data so old clients can still talk&listen&message with new clients.)
Okay, I updated my mock up a bit, what I did was change where Meta Opcode and Call Opcode was, renamed Call Opcode Payload Opcode, and trimmed Payload Opcode to 3 bits, and the Most Significant Bit of the TYPE field should always be 1 to distinguish between the newer LSF TYPE field, and the older 2.0 and earlier style.
I don't think there is any reason to devote a bit to Version 3. Just reserve 0x0 in the Payload Op. so that it will never be used. I would like to see 15 possible values because we will immediately need at least 6 values for for payloads we already have: data-only, C2-1600+data, C2-3200, C2+1600-Signed, C2-3200-signed, and packet.
I'd like to leave room for some other codecs. Maybe there's some 40ms, 128-bit open-source codec out there that can approach 16-bit, 8000 Hz PCM that's just waiting to be discovered.
I finally had a chance to review this. Some comment/questions:
- @lwvmobile Perhaps you could paste your proposal text in block-formatted code, that is starting and ending with ``` lines. That would allow others to cut and paste, make changes, etc.
- I like the way this is looking, particularly the approaches to compatibility. In addition to @n7tae's idea of transcribing between V2 and V3, I could do something similar in
m17-gateway. If new versions of OpenRTX can still receive V2, then the hotspot/repeater could transcribe V3 to V2, allowing radios to interoperate until they all are upgraded. - I'm not sure what Opcode means in this context. Is it just a synonym for format?
- Does the Encryption field apply to all Data Opcodes? In the old TYPE format, Encryption Type/Subtype only applied to stream format (byte 1, bit 0 = 1) frames.
- How do Data Opcode and Meta Opcode interact? Are they orthogonal or do Meta Opcodes vary for different Data Opcodes? For example, in the old TYPE format, Encryption Type/Subtype only applied to stream format (byte 1, bit 0 = 1) frames so that GNSS, Meta Text and Extended Callsign Data officially only applied to stream data. Is that still the case? I'd vote for making them orthogonal, but some might only make sense for particular data opcodes.
I don't think there is any reason to devote a bit to Version 3. Just reserve 0x0 in the Payload Op. so that it will never be used. I would like to see 15 possible values because we will immediately need at least 6 values for for payloads we already have: data-only, C2-1600+data, C2-3200, C2+1600-Signed, C2-3200-signed, and packet.
Oh, Okay, I see what you mean know. Probably works out better that way, allows more opcodes. Any opcode value I used are only as examples in the mock up, so they can be any value, skipping zero to maintain V2 compatibility.
- Perhaps you could paste your proposal text in block-formatted code, that is starting and ending with ``` lines. That would allow others to cut and paste, make changes, etc.
I'll try to do that for the next little fix I make to address above.
I'm not sure what Opcode means in this context. Is it just a synonym for format?
Pretty much, its just a catch-all term, but here means that if the opcode is a value, the associated payload or META only contains this content.
- Does the Encryption field apply to all Data Opcodes? In the old TYPE format, Encryption Type/Subtype only applied to stream format (byte 1, bit 0 = 1) frames.
In my proposal it does to keep the "keep it simple" approach. I don't see a particular reason for the TYPE field to have different meaning for data and stream, lacking fields on data that are there for the other. I realize encryption is a topic that is usually frowned upon, but data packets could also be encrypted.
- How do Data Opcode and Meta Opcode interact? Are they orthogonal or do Meta Opcodes vary for different Data Opcodes? For example, in the old TYPE format, Encryption Type/Subtype only applied to stream format (byte 1, bit 0 = 1) frames so that GNSS, Meta Text and Extended Callsign Data officially only applied to stream data. Is that still the case? I'd vote for making them orthogonal, but some might only make sense for particular data opcodes.
In my proposal, they do not interact together. They do not vary. That was the point. The contents of META do not need to rob bits from the encryption field depending on the context of using encryption or not using it. Another user mentioned a while back they wanted GNSS ability in Packet Data, this seperation will allow doing so. Again, the idea is that the LSF between packet data and stream should be the same, in my opinion.
LSF Type Field (4-bit alignments)
-----------------------------
| Payload Op | Meta Opcode |
-----------------------------
| Encryption | CAN |
-----------------------------
Changing the LSF's 16-bit type field will be a breaking change, as such, it may be prudent to future proof the Type Field by allowing for more potential, particularly in the Meta field, and also call types. This Type Field can be used for both Stream Sync and Packet Data Sync patterns.
CAN: 4-bit value, as-is, now bit/byte aligned.
Encyption: Encryption Type and Subtype Field. Meta Contents are removed from this field and placed into Meta Opcode Field. If using AES, Meta Opcode needs to be set to "Meta IV" Opcode Value.
Payload Opcode: Specifies what is in the data type field, and also includes the signed stream bit, or new opcode values can be made to incorporate 3200 voice, 1600 voice + data, either with or without signed stream, and perhaps other data call types, if desired. Packet Data should be inferred by the sync pattern, but can also be given a unique Call Opcode, if desired.
Example Opcodes: 0x0 - UNUSED - Version 2.0 Backwards Compatibilty Reserved Bits Field
0x1 - 3200 Voice
0x2 - 1600 Voice + Arbitrary Data
0x3 - 3200 Voice + Signed Stream
0x4 - 1600 V+D + Signed Stream
0x5 - Packet Data
...
0xF - ???
Meta Opcode: Specifies the Data that is found in the Meta Data field. For ease of use, carrying over the current "encryption subtype" values as opcodes for GNSS Data, Text Data, Extended Callsign Data as-is into this field, and allowing new Meta Opcode types such as Meta Encryption IV, Meta User Properties, and similar.
Example Opcodes: 0x0 - NULL Data
0x1 - Text Data
0x2 - GNSS Position Data
0x3 - Extended Callsign Data
0x4 - Extended User Properties (n7tae suggestion)
...
0xF - AES Encryption IV
I'm going to issue an update to my last suggestion, mostly regarding nomenclature, changing the word "Opcode" to "Contents" at @sp5wwp 's suggestion.
Also, signed stream can be moved from the Payload Contents Element to the Encryption Option Element if we merge Encryption Type and Encryption Subtye into a single 4-bit field, the MSB (1000) will signify Signed Stream, while the three lower bits will be encryption types already specified in M17 Specification. Removing Signed Stream for Payload Contents Element will free up even more content values for future use cases.
LSF Type Field (4-bit alignments)
----------------------------------------
| Payload Contents | Meta Contents |
----------------------------------------
| Encryption Option | CAN |
----------------------------------------
Changing the LSF's 16-bit type field will be a breaking change, as such, it may be prudent to future proof the Type Field by allowing for more potential, particularly in the Meta field, and also call types. This Type Field can be used for both Stream Sync and Packet Data Sync patterns.
CAN: 4-bit value, as-is, now bit/byte aligned.
Encyption Option: Just one value now for Encryption Type and Subtype. Meta Contents are removed from this field and placed into Meta Contents Field. If using AES, Meta Contents needs to be set to "Meta IV" Contents Value. Signed Stream bit is the MSB of this element, to OR 0x8 onto (1000) and the three LSB are encryption ciphers currently included into M17 Specification.
Example Contents: 0x0 - No Encryption
0x1 - Scrambler 8-bit
0x2 - Scrambler 16-bit
0x3 - Scrambler 24-bit
0x4 - AES 128-bit
0x5 - AES 192-bit
0x6 - AES 256-bit
0x7 - Reserved
0x8 - (MSB) OR (|=)onto Encryption Field for Signed Stream
Payload Contents: Specifies what is in the data type field, and also includes the signed stream bit, or new Contents values can be made to incorporate 3200 voice, 1600 voice + data, either with or without signed stream, and perhaps other data call types, if desired. Packet Data should be inferred by the sync pattern, but can also be given a unique Call Contents, if desired.
Example Contents: 0x0 - UNUSED - Version 2.0 Backwards Compatibilty Reserved Bits Field
0x1 - 3200 Voice
0x2 - 1600 Voice + Arbitrary Data
0x3 - Packet Data
...
0xF - ???
Meta Contents: Specifies the Data that is found in the Meta Data field. For ease of use, carrying over the current "encryption subtype" values as Contentss for GNSS Data, Text Data, Extended Callsign Data as-is into this field, and allowing new Meta Contents types such as Meta Encryption IV, Meta User Properties, and similar.
Example Contents: 0x0 - NULL Data
0x2 - GNSS Position Data
0x3 - Extended Callsign Data
0x4 - Extended User Properties (n7tae suggestion)
0x5 - Text Data
...
0xF - AES Encryption IV
@sp5wwp Also wanted me to mention, that this allows for the inclusion of other META types to be used if encryption is enabled. As-is, Encryption META is only required as an IV for AES, but that isn't required to be transmitted every superframe. It can also be round-robin, since the 114-bit IV transmitted in META is static, AES is operated in CTR mode, and uses the frame number as its counter. AES IV only needs to lead off on the LSF frame and first superframe, and be signaled once every several superframes in the case of late joiners to the call and those who missed the initial LSF framesync pattern.
Scrambler does not use the META field, and can be used freely under this proposal while transmitting any other contents via META.
AES IV only needs to lead off on the LSF frame and first superframe
LSF - yes, it's a must. First superframe - IV doesn't necessarily need to be there, I think.
Should also mention, the more you load into the META field for round-robin, the longer a transmission will need to be in order to convey all that information. While META is a secondary type of information to send on stream frames particularly, I see a lot of short burst on the reflector, so if you load up a GNSS position, a long META Text Message, and so on, the longer the PTT has to be to get all of that out there, assuming its of enough importance to the user to make sure the other end always receives all of that data.
Also, to alleviate any concerns, this won't be an overnight flip of the switch and hard cut off, the 3.0 spec is in a dev branch and we will all transition to it over time, and code for backwards compatibility while maintaining that version 2.0 will be the primary and fallback version for the time being.
This is getting very close. I have a few suggestions to finalize...
Please put all the payload stuff in one byte by swapping Meta with encrypt. And let's call it like it is, Encrypt is 3 bits and signing is one bit. It's more of a mnemonic, but I think signing should be the least significant bit of the first byte, just like a signature is at the end of a letter, so for the first byte, bits 7..4 is payload type, 3..1 is encrypt type and 0 is the signing bit. And how about 0xF for packet payload, leaving 0x4..0xE for new payload types? Also you should probably leave User Properties out of Meta for now, too much has to be decided to get that working, and it can be added later as a minor revision (new feature) if there is a consensus that it's a good idea and will be fully supported.
I am anxious to see the V3 spec finished. I have started working on a new version of mrefd that will introduce a new client type. It will be identical to the existing M17 client, but it will use a CON3 connection packet instead of a CONN connection packet. CON3 clients will use data packets that use the new V3 FrameType format. Mrefd will then make sure that each client will receive data appropriate for its requirement. Legacy clients will be able to continue to connect to the new mrefd and they will also be able to talk/listen/text to V3 clients. Then everyone can move to version 3 when they choose to do so. This will be a breaking change for mrefd, but only for reflectors that do reflector-reflector interlinking, otherwise it's just a simple upgrade. Client developers can then also make clean upgrades to V3 without having to support V2 as well, if they choose.
I am anxious to see the V3 spec finished. I have started working on a new version of mrefd that will introduce a new client type. It will be identical to the existing M17 client, but it will use a
CON3connection packet instead of aCONNconnection packet.
Are you going to have a LSN3 to go with it, some of us don't have a license yet, but I still like to be able to listen with the V3, if possible.
Please put all the payload stuff in one byte by swapping Meta with encrypt. And let's call it like it is, Encrypt is 3 bits and signing is one bit. It's more of a mnemonic, but I think signing should be the least significant bit of the first byte, just like a signature is at the end of a letter, so for the first byte, bits 7..4 is payload type, 3..1 is encrypt type and 0 is the signing bit. And how about 0xF for packet payload, leaving 0x4..0xE for new payload types?
I think I hit all the points you mentioned, if not, let me know and I'll try again.
LSF Type Field (4-bit alignments)
----------------------------------------
| Payload Contents | Encryption / Sig |
----------------------------------------
| Meta Contents | CAN |
----------------------------------------
Changing the LSF's 16-bit type field will be a breaking change, as such, it may be prudent to future proof the Type Field by allowing for more potential, particularly in the Meta field, and also call types. This Type Field can be used for both Stream Sync and Packet Data Sync patterns.
CAN: 4-bit value, as-is, now bit/byte aligned.
Encyption Option: Just one value now for Encryption Type and Subtype. Meta Contents are removed from this field and placed into Meta Contents Field. If using AES, Meta Contents needs to be set to "Meta IV" Contents Value.
Signed Stream bit is the least significant bit of this element (bit 0), and the three most significant bits (bits 3,2,1) are encryption ciphers currently included into M17 Specification.
Example Contents: (bit 0)
0x0 - Not Signed Stream
0x1 - Signed Stream
(bits 3,2,1)
0x0 - No Encryption
0x1 - Scrambler 8-bit
0x2 - Scrambler 16-bit
0x3 - Scrambler 24-bit
0x4 - AES 128-bit
0x5 - AES 192-bit
0x6 - AES 256-bit
0x7 - Reserved
Payload Contents: Specifies what is in the data type field, and also includes the signed stream bit, or new Contents values can be made to incorporate 3200 voice, 1600 voice + data, either with or without signed stream, and perhaps other data call types, if desired. Packet Data should be inferred by the sync pattern, but can also be given a unique Call Contents, if desired.
Example Contents: 0x0 - UNUSED - Version 2.0 Backwards Compatibilty Reserved Bits Field
0x1 - 3200 Voice
0x2 - 1600 Voice + Arbitrary Data
...
0xF - Packet Data
Meta Contents: Specifies the Data that is found in the Meta Data field. For ease of use, carrying over the current "encryption subtype" values as Contents (mostly) for GNSS Data, Extended Callsign Data as-is into this field (Meta Text is now 3, allowing 0 to be NULL data), and allowing new Meta Contents types such as Meta Encryption IV, Meta User Properties, and similar.
Example Contents: 0x0 - No Contents (Zero Fill)
0x1 - GNSS Position Data
0x2 - Extended Callsign Data
0x3 - Text Data
...
0xF - AES Encryption IV