CBORObject.read Map issue

[ShanmukhVegi]

Hi,

We are using CBOR v4.0.1 currently and we read the data received from our server using CBORObject.read(stream); which returns a list of map and had our use-case fulfiled.

Now, we tried to upgrade to v4.5.5 and then we're seeing an issue with the order of the map not being preserved, meaning in v4.0.1 in the map object we use to get {"K1":"V1", "K2","V2"} and after updating to v4.5.5 we are getting {"K2":"V2","K1":"V1"} (Note: This is not happening with all the keys).

I checked the documentation and figured out in v4.5.0, an option to preserve the map order is added in CBOREncodeOptions and after this, I updated my code to CBORObject.read(stream, new CBOREncodeOptions("keepkeyorder=true")) and this is working as expected.

So in v4.0.1, is the order of the map preserved by default (like having keepkeyorder to true)?

[peteroupc]

I believe not.

Indeed, the EncodeToBytes method documentation says: "If the CBOR object contains CBOR maps, or is a CBOR map itself, the order in which the keys to the map are written out to the byte array is undefined unless the map was created using the NewOrderedMap method." The KeepKeyOrder option decodes CBOR maps "by using maps created as though by CBORObject.NewOrderedMap".

[ShanmukhVegi]

Hey, thanks for the swift response!

Okay, I believe we didn't use NewOrderedMap as it was introduced in v4.3. Is there any logic change between v4.0.1 and v.4.5.5 for CBORObject.read() method? I ran the test for about 18 different cases and collected the CBOR read response given by server when using v4.0.1 and checked the response of same sample running against v4.5.5.

The table below contains the following:

• CBOR content: Actual CBORObject (CBOR type: Map)
• 4.0.1 : Response read after receiving from server using CBORObject.read(stream); in v4.0.1
• 4.5.5 (false) : Response read after receiving from server using CBORObject.read(stream, new CBOREncodeOptions("keepkeyorder=false")) in v4.5.5
• 4.5.5 (true) : Response read after receiving from server using CBORObject.read(stream, new CBOREncodeOptions("keepkeyorder=true")) in v4.5.5

CBOR content	4.0.1	4.5.5 (false)	4.5.5 (true)
{"orgId": "N", "orgName": "S"}	Same	Same	Same
{"myId": "S", "SortKey": "S"}	Same	Same	Same
{"PartitionKey": "S", "SortKey": "S"}	Same	{"SortKey": "S", "PartitionKey": "S"}	Same
{"PK": "S", "SK": "S"}	Same	Same	Same
{"myPk": "S", "mySk": "S"}	Same	Same	Same
{"PartitionKey": "S", "sk": "S"}	Same	{"sk": "S", "PartitionKey": "S"}	Same
{"pk": "S", "SortKey": "S"}	Same	Same	Same
{"OrganizationId": "S", "SortKey": "S"}	Same	{"SortKey": "S", "OrganizationId": "S"}	Same
{"OrganizationId": "S", "Sk": "S"}	Same	{"Sk": "S", "OrganizationId": "S"}	Same
{"mypkID": "S", "mySK": "S"}	Same	{"mySK": "S", "mypkID": "S"}	Same
{"mypk": "S", "mysk": "S"}	Same	Same	Same
{"myId": "S", "sk": "S"}	Same	{"sk": "S", "myId": "S"}	Same
{"endingId": "S", "sk": "S"}	Same	{"sk": "S", "endingId": "S"}	Same
{"myPk": "S", "sk": "S"}	Same	{"sk": "S", "myPk": "S"}	Same
{"pk": "S", "sk": "S"}	Same	Same	Same
{"pkk": "S", "sk": "S"}	Same	{"sk": "S", "pkk": "S"}	Same
{"p": "S", "sk": "S"}	Same	Same	Same
{"wd": "S", "s1": "S"}	Same	{"s1": "S", "wd": "S"}	Same

So, here it can be seen that when using CBORObject.read(stream, new CBOREncodeOptions("keepkeyorder=true")) in v4.5.5, I'm getting the exact same output as of using CBORObject.read(stream) in v4.0.1

[peteroupc]

Even in version 4.0.1:

• The EncodeToBytes() method documentation said: "If the CBOR object contains CBOR maps, or is a CBOR map itself, the keys to the map are written out to the byte array in an undefined order."
• The three-parameter Write method documentation said: "If the object is convertible to a CBOR map or a CBOR object that contains CBOR maps, the keys to those maps are written out to the data stream in an undefined order.

That version 4.0.1 may have preserved the order of map keys was merely an accident of the implementation and was not to be relied on by applications using that version.

[ShanmukhVegi]

The v4.0.1, we have been using for multiple years and didn't face any issues.

In v4.0.1, I see the NewMap() implementation is using Simple Dictionary (ref: https://github.com/peteroupc/CBOR/blob/v4.0.1/CBOR/PeterO/Cbor/CBORObject.cs#L2314)

And in v4.5.5, I see the NewMap() implementation is using SortedDictionary (ref: https://github.com/peteroupc/CBOR/blob/v4.5-branch/CBOR/PeterO/Cbor/CBORObject.cs#L3309)

So, is this causing an issue while reading the data in v4.5.5 and not in v4.0.1?

[peteroupc]

The underlying implementation of Dictionary in your .NET runtime may have preserved the insertion order of map keys. However, my CBORObject NewMap documentation doesn't make such a guarantee, and neither does System.Collections.Generic.Dictionary.

By version 4.5.5, the underlying implementation of NewMap was changed in the meantime to SortedDictionary, which sorts CBOR objects according to their "lexicographical order". In the case of text strings, though, "lexicographical order" is not necessarily "alphabetical order", since, for example, text strings with a smaller "byte size" are sorted first. This can be seen in the third column of the table, where "shorter" keys come before "longer" keys.

Again, however, the order in which CBOR map keys are written (whether insertion order or "lexicographical order") is documented to be undefined, with certain exceptions. And, again, that the insertion order was preserved in your case in version 4.0.1 is only an implementation detail in whatever implementation of .NET your application was relying on.

[peteroupc]

No further response seen, so closing.