Binary protocol with ASCII characters

[abrasat]

We have to exchange data with some external partner which uses a binary protocol with ASCII characters and I am not sure if the de-/serialization can be achieved with the help of the BinarySerializer. For instance all value fields are separated with the ";" delimiter (ASCII code 0x3B). For example: class SampleData1 { string id; string value1; int value2; int value3; } initialized for instance with id = "ID1"; value1 = "abc"; value3 = 7; (value 2 on purpose not initialized->default value should be used). According to the required protocol a string can have the meaning "string value" (in this case it is delimited with the string identifier " = 0x22) or "string identifier" (transported without string identifier).
The class above would be binary serialized as (in ASCII printable representation, not hex):

ID1;"abc";;7;

To get things even more complicated, lists (arrays) are also supported. The arrays are delimited with "[" and "]" and the array elements are separated with ",". The aggregate types (classes) are additionally delimited with "{" and "}".

For example an array of 2 SampleData2 elements (the number of array elements is dynamic, and should be determined by the BinarySerializer by recognizing the "[" and "]" delimiters), where SampleData2 is defined as:

class SampleData2 { SampleData1 data1; int var1; }

would be serialized to (in ASCII printable representation) as: [{{ID1;"abc";;7;};3},{{ID2;"def";;;};44}]

(in this example var1 is 33 in the first array element and 44 in the second)

The classes and arrays can be hierachically cascaded (that means classes may contain other classes or arrays and so on).

Would be possible to implement such a protocol with BinarySerializer ? Are there any samples for similar protocol available ?

[jefffhaynes]

Craziness!  Ok, let me think about it but it might have to wait until the weekend.  Thanks, Jeff

On Wed, Jan 27, 2016 at 5:26 AM -0800, "abrasat" [email protected] wrote:

We have to exchange data with some external partner which uses a binary protocol with ASCII characters and I am not sure if the de-/serialization can be achieved with the help of the BinarySerializer

For instance all value fields are separated with the ";" delimiter (ASCII code 0x3B) For example:

class SampleData1

{

string id;

string value1;

int value2;

int value3;

}

initialized for instance with

id = "ID1";

value1 = "abc";

value3 = 7;

(value 2 on purpose not initialized->default value should be used)

According to the required protocol a string can have the meaning "string value" (in this case it is delimited with the string identifier " = 0x22) or "string identifier" (transported without string identifier)

The class above would be binary serialized as (in ASCII printable representation, not hex):

ID1;"abc";;7;

To get things even more complicated, lists (arrays) are also supported The arrays are delimited with "[" and "]" and the array elements are separated with "," The aggregate types (classes) are additionally delimited with "{" and "}"

For example an array of 2 SampleData2 elements (the number of array elements is dynamic, and should be determined by the BinarySerializer by recognizing the "[" and "]" delimiters), where SampleData2 is defined as:

class SampleData2

{

SampleData1 data1;

int var1;

}

would be serialized to (in ASCII printable representation) as:

[{{ID1;"abc";;7;};3},{{ID2;"def";;;};44}]

(in this example var1 is 33 in the first array element and 44 in the second)

The classes and arrays can be hierachically cascaded (that means classes may contain other classes or arrays and so on)

Would be possible to implement such a protocol with BinarySerializer ? Are there any samples for similar protocol available ?

— Reply to this email directly or view it on GitHub.

[abrasat]

Great, thanks! I hope this will not discourage you :-), but I "discovered" two more specialties in the protocol. The first is a kind of dictionary (or better said an array of key value pairs). The key and the value are separated by the ":" character (0x3A), and the key-value pairs are separated by "," (actually as between any two array elements). For instance: [(Key1:1),(Key234:3),(ABC:77)]

The second is the possibility to send "raw" byte arrays (that means the content must be interpreted as raw hexadecimal code instead of ASCII character code which is then converted to the corresponding number or string) These are marked by the "$" character in front of the raw hex byte values, for example: $0102030405060708090A

[jefffhaynes]

I've been thinking about this and one of my concerns is that there are a lot of potential issues you run into. For example, what happens if a semicolon appears in a string? Is there an escaping sequence? If so, at some point it starts to feel more like a text formatter/parser than straight serialization/deserialization.

[abrasat]

Actually there is no need of escaping sequence, as none of the used delimiters are allowed to appear as valid characters inside strings (but of course this is a convention, and the BinarySerializer has to handle it in a general way). The protocol described above is some kind of mixture between binary serialization and text formatter/parser. The delimiters (start-delimiter of array, end-delimiter of array, separator between array elements) can be also seen as some modality to describe arrays with dynamic sizes, regardless if these delimiters are ASCII characters or not. Of course, as described above, the delimiters are not allowed to appear as valid data, otherwise this would lead to serialization exceptions. But I see that this could mean a lot of data validation handling inside the BinarySerializer. Deserializing the binary data into one "big" string and parse it further as a text could also be a pragmatic approach to solve the above problem.

[jefffhaynes]

I know this is really old but I just pushed 7.6, which allows for arbitrary string termination if that helps.