Sequences with end marker

[treiher]

Examples are the options fields of IPv4 and CoAP. The end marker is only present in IPv4, if the end of the last option does not match the end of the options field. The end marker of CoAP is outside of the actual options field.

The end marker of a sequence could be defined by an aspect of the sequence type declaration. Maybe an additional aspect is needed to define, if the end marker must be present. The end marker itself could be represented by an atomic type (or a message type).

[hurufu]

Thanks for releasing such great tool. I have a protocol that is used over a serial connection and is based on ASCII text delimited by control characters (STX,ETX,FS,US). I hope this task will be implemented soon.

[senier]

Hi Alex, thanks! We haven't scheduled this feature for implementation, yet (mostly, because very few protocols require it). It would be interesting to know a bit more about your use case. The "ASCII text" part sounds a bit like RecordFlux may not be the right tool here. While we see a need for mixed binary / text protocols (e.g. for DNS), we have no plans to support text protocols anytime soon. You may get away with modelling the text using the Opaque type, but that may not be very useful.

[hurufu]

@senier Message format is quite simple, but the protocol has arcane rules describing when each field must be present or absent because they depend on each other. An example message would be something like: <STX>123<FS>abc<FS><ETX> (with LRC at the end), most fields are small – up to 20 bytes with constant size, but some of them have variable size that's why I was thinking if this "end marker" can be useful. Right now I have a manually written parser, I just googled if there is any tool that can be useful here.

[senier]

@hurufu This is really different from what we have modeled so far (and yes, it looks a bit arcane :wink: ) If there is no length field, you'll definitely need the end marker. For the ASCII control characters you also might want constant fields (#63) -- it's possible without but the specification might become pretty ugly without.

Here is how such a specification could look like if had end markers (I'm just making the end marker syntax up as I go -- the Until aspect is not valid):

package Protocol is

   type Character is range 0 .. 255 with Size => 8;
   type String is sequence of Character;

   type Message is
      message
         STX : Character
            if STX = 2;
         Name : String
            with Until => Name'Element = 28; -- FS
         Value : String
            with Until => Value'Element = 28; -- FS
         ETX : Character                                                                                                                                                      
            if ETX = 3;
         LRC : Character
            if LRC'Valid_Checksum;
      end message
         with Checksum => (LRC => (Name, Value));

end Protocol;

That's for a single message. Not sure which dependencies you'd want to express (between fields? between messages?).

[hurufu]

Cool! That's exactly how I'd imagined it. Dependencies are between fields, so using your suggested syntax an optional field may be written like this (I'm not well versed in RFLX yet, but I've tried to make it more-or-less consistent)

package Protocol is

   type Byte is mod 256 with Size => 8;
   type ControlCharacter is range 16#00 .. 16#20# with Size => 8;
   type SafeCharacter is range 16#21# .. 16#7F# with Size => 8;
   type String is sequence of SafeCharacter;

   type Message is
      message
         StartFrame : ControlCharacter
            if StartFrame = 2;
         Name : String
            with Until => ? -- FS was encountered
            then EndFrame
                if Name'Length = 0;
         Value : String
            with Until => ? -- FS was encountered
         EndFrame : ControlCharacter                                                                                                                                                      
            if EndFrame = 3;
         LRC : Byte
            if LRC'Valid_Checksum;
      end message
         with Checksum => (LRC => (Name, Value));

end Protocol;

Here I was trying to describe a message that has 2 fields. 1st (Name) is mandatory and 2nd (Value) is conditional. If 1st field is empty then second field must be absent, otherwise 2nd field should be present. Note also that String can include only 7bit ASCII without control characters, but LRC can be any byte.

Does it make sense?

[senier]

@hurufu You're raising some interesting questions here :wink:

We played around with that design idea a bit. What seemed a little awkward is that the field separators after Name and Value are not really represented in the spec (just implicitly in the Until aspect). Also, using Name'Length in the condition for Name felt confusing. What if we haven't read Name completely yet - is the length valid then?

We found it beneficial to introduce dedicated types to make field separators explicit and control the length of the strings by their types. Those fields would not have any condition or length anymore, but consume elements until there is an element that is outside the domain of the element type. This would allow a pretty natural and concise description:

package Protocol is

   type Byte is range 0 .. 255 with Size => 8;
   type Character is range 32 .. 127 with Size => 8;
   type String is sequence of Character;

   type STX is range 2 .. 2 with Size => 8;
   type ETX is range 3 .. 3 with Size => 8;
   type FS  is range 28 .. 28 with Size => 8;

   type Message is
      message
         Start_Frame : STX;
         Name : String;
         Name_Separator : FS
            then EndFrame
                if Name'Length = 0
            then Value
                if Name'Length > 0;
         Value : String;
         Value_Separator : FS;
         End_Frame : ETX;
         LRC : Byte
            if LRC'Valid_Checksum;
      end message
         with Checksum => (LRC => (Name, Value));

end Protocol;

Note, that we introduced specific types for STX, ETX and FS which allow only values for that one control character, respectively. To make Value optional, you could take different paths after Name_Separator, depending on the length of Name.

Such a solution would require us to analyze the structure of the message to make sure that after an unbounded sequence there comes a field which allows only values outside the domain of the sequence elements. This could lead to some interesting corner cases, but the concise specification is probably worth the effort.

Would that accurately describe your scenario?

[hurufu]

I played a little bit with this idea and I think it will work. Thanks a lot. Obviously I don't expect this feature any time soon, but still a lot of respect for it.

P.S. Happy New Year! P.P.S. As a side note specification can be found here, but I don't expect you to read it.

[NickFoubert]

+1

I believe this feature would be needed to specify parts of the Real Time Publish-Subscribe (RTPS) wire protocol which I'm trying to do right now!

https://www.omg.org/spec/DDSI-RTPS/2.5/PDF

For example, a ParameterList (section 9.4.2.11 in the linked spec) field uses a sentinel (end marker) in most cases.