gofast

gofast copied to clipboard

This implementation assigns tags 37-39 with different meanings than in the document http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml#tags . Using these tags should be avoided, as CBOR data that makes use of them with the meanings other than specified in that document would make them less interoperable; instead, tags starting from 266 should be used; they're currently not assigned according to that document.

Hello Peter, thanks for the notification.

gofast is still experimental, so it is not too late to change the tag values. And I would like to make gofast fully CBOR compliant. You might have noticed that tag values,

• 6,7,8 are used for protocol framing.
• 37,38,39,40,41,42 are used for payload description.

protocol framing tags 6,7,8 need to be single byte because it is very useful to debug transport exchange between nodes while authoring client SDK in different languages.

among the {37,38,39,40,41,42} set of description tags,

• I can shift 38, 41, 42 after 266.
• But I would prefer to shift 37,39,40 into 40-255 unassigned space, since every packet will carry those tag - I would like to keep them optimized for "internet of things".

Let me know what you think.

It would be awesome if CBOR community would consider gofast, a fully CBOR compliant, bi-directional / peer-peer streaming protocol as a open standard. If there is anything that I can do to that end, please let me know.

Thanks,

The issue here is that the value space from 0-255 is very scarce, so they require stricter procedures for assigning codes to that space. For more information, see section 7.2 of RFC 7049 and section 4.1 of RFC 5226.

• Values from 0-31 require Standards Action, that is, a Standards Track RFC approved by the Internet Engineering Steering Group (IESG).
• Values from 32-255 use a Specification Required policy.
• Values 256 and greater use a First Come First Served policy.

After your decision, I can say that:

• Codes 37 and 39 still conflict with the "binary UUID" and "Identifier" tags, respectively. For tag 37, you may consider using tag 24 instead, if the encoded payload is in CBOR.
• Tags 6 to 8 can't simply be used as you propose without a Standards Track document approved by the IESG, which I think is unlikely due to the highly specialized nature of tags 6-8 as they are currently used. Namely, the other tags in the space 0-31 are general-purpose data types, but the three tags as you use them are just message codes used in one specific protocol.

You need to move tags 6 to 8 and tag 37 and 39 elsewhere in the space to avoid conflicts. If possible, you should also check whether any of the existing tags in the registry meet your needs, and merge certain new tags into one tag to avoid unnecessary tag codes. Visit http://www.iana.org/protocols/apply to apply to add new tags to the CBOR tag registry, and include the information requested in section 7.2 of RFC 7049 in the application.

I guess I can finally make up my mind on this issue. It stems from the following claim from gofast's README page:

Reading a frame from socket

Framing of packets are done such that any gofast packet will at-least be 9 bytes long. This is because encoding payload and packet takes at-least 5 bytes and framing takes 4 bytes.

Incidentally these 9 bytes are enough to learn how many more bytes to read from the socket to complete the entire message.

To keep the framing bytes constant at 4 bytes I have to use one of the reserved tags. It is not only simplifying the reader logic, it also gives a performance advantage of just reading twice from the socket to get the complete gofast-message.

Having said this, I will still keep this issue open so that we can track the pros and cons of this decision.

If there are any immediate concerns please let me know.

Thanks,

http://www.iana.org/assignments/cbor-tags/cbor-tags.xhtml#tags

As I said before, you need to assign CBOR tags so they don't conflict with existing tags. I will discuss each in turn:

• Tags 6, 7, 8 -- To use these tags in your protocol, you should have a standards-track RFC approved by the IESG. Unfortunately, moving these to tag numbers greater than 23 will take up more space. Fortunately, all three tags are still unassigned.
• Tag 37 -- Conflicts with the Binary UUID tag; move it to tag 31, tags 42-95, or 99-255, which will take the same space as tag 37.
• Tag 38 -- Conflicts with the Language-Tagged String tag; move it to tag 31, tags 42-95, or 99-255, which will take the same space as tag 38.
• Tag 39 -- Conflicts with the Identifier tag; use tag 24 instead or move it to tag 31, tags 42-95, or 99-255, which will take the same space as tag 39.
• Tags 40, 41 -- These tags are unassigned and seem like good candidates for IANA registration, since they seem not to be specific to this protocol. Again, visit http://www.iana.org/protocols/apply to apply to add new tags to the CBOR tag registry.

@cabo : Please give your thoughts.

So why don't we write this protocol up and get a "specification required" assignment?

I'm not so hot about the single-byte tags; we have precious few of them and would like to reserve them for very general, heavily used tags with short data items. I believe you would be very well served with the specification required tags.

I don't understand tag 8 (current number) -- what is this being applied to?

Thanks @peteroupc, @cabo.

I have followed your suggestion regarding Tag-37 to Tag-41 (inclusive). I am moving them to Tag-43 to Tag-47.

Regarding Opaque-value:

By design opaque value should be >= 256. These are ephimeral tag values that do not carry any meaning other than identifying the stream. Opaque values will continuously reused for the life-time of connection. Users are expected to give a range of these ephemeral tag-values, and I can make gofast to skip reserved TAGS.

@cabo question about Tag-8 made me to look into the end-packet framing. And figured that is not entirely in line with what I wanted it to be. I am fixing it as follows:

    | len | tag1  | 0x40 | 0xff |

• End-packet does not carry any useful payload, it simply signifies a stream close.
• For that, tag1 opaque-value is required to identify the stream.
• 0x40 is the single-byte payload for this packet, which says that the payload-data is ZERO bytes.
• The last 0xff is important since it will match with 0x9f that indicates a stream-start as Indefinite array of items.

Regarding Tags-6,7,8:

After some experiments, found that the overhead of using gofast protocol to exchange messages between nodes can be at-least 24 bytes. If I move Tags-6,7,8 to specification-required space there might be additional 1/2 byte overhead, which I believe is okay.

That leaves only one more technicality:

Streaming messages

Following is the byte-dump of gofast-frames, carrying a stream of 3 messages, each with payload of 14 bytes, the last message is followed by end-packet.

[d9, d9, f7, 9f, 58, 1e, d9, 3e, 81, 58, 19, d8, 2b, bf, d8, 2c, 18, 71, d8, 2d, 4e, 0, 0, 0, 0, 0, 0, 0, 6, 61, 61, 61, 61, 61, 61, ff] [d9, d9, f7, c7, 58, 1e, d9, 3e, 81, 58, 19, d8, 2b, bf, d8, 2c, 18, 71, d8, 2d, 4e, 0, 0, 0, 0, 0, 0, 0, 6, 61, 61, 61, 61, 61, 61, ff] [d9, d9, f7, c7, 58, 1e, d9, 3e, 81, 58, 19, d8, 2b, bf, d8, 2c, 18, 71, d8, 2d, 4e, 0, 0, 0, 0, 0, 0, 0, 6, 61, 61, 61, 61, 61, 61, ff] [d9, d9, f7, c8, 44, d9, 3e, 81, 40, ff]

deciphering first frame:

• [d9, d9, f7] CBORPrefix
• [9f] stream start, as indefinite array of item.
• [58, 1e] - byte-string of length 30. Also, first item that is part of indefinite array.

Automatically second and third frames act as the second and third item of the indefinite array.

deciphering the fourth frame end-packet:

• [d9, d9, f7] CBORPrefix
• [c8] tag-8 denotes end-packet
• [44] byte-string of length 4.
  • [d9, 3e, 81] opaque-value that identifies the stream, ephemeral tag.
  • [40] byte-string of zero-length.
• [ff] BreakStop, complete the indefinite array.

So the fourth frame behaves as the fourth item in the indefinite array terminated by break-stop.

we have precious few of them and would like to reserve them for very general, heavily used tags with short data items.

Please note that we are more or less building a protocol similar to HTTP-2 which can be specified in less than 5 pages. Thanks to CBOR.

Gofast already supports HTTP functions like:

• Concurrent streams on same connection, aka multiplexing.
• It models all types of exchanges (programmatically speaking) that are possible over network - POST, REQUEST, STREAM.
• Bi-directional STREAM-ing.
• Peer-to-peer, either node can POST,REQUEST,STREAM data.
• Versioning and Settings are not part of the packet-framing but are included as part of the initial handshake.
• Headers as map[tag]CBOR-data.

It would be nice if Gofast can retain Tag-6,7,8. That is, if we think Gofast can cater to many applications.

I cannot really decode these messages. In particular, the fourth message ends in an ff that is not closing any indefinite header (9f, bf) in the same message.

One very useful tool here is http://cbor.me — here you can paste messages in hex (commas are ignored) on the right side and use the arrow to decode them; the inverse direction also works.

Concatenating all 4 gofast-frames and diagnostic o/p from http://cbor.me

55799([h'D93E815819D82BBFD82C1871D82D4E0000000000000006616161616161FF', 55799(7(h'D93E815819D82BBFD82C1871D82D4E0000000000000006616161616161FF')), 55799(7(h'D93E815819D82BBFD82C1871D82D4E0000000000000006616161616161FF')), 55799(8(h'D93E8140'))])

0xff matches 0x9f from the first-frame.