Request: getByteArray to retrieve a sequence of octets (CON-42)

[BrianAdams]

We are passing around large octet sequences. The good news is that we can access the data through the getJson API. The data comes across as an array of strings that represent the bytes. Thus to get the actual byte Array we have to map the resulting values like this:

var stringdata = input.samples.getJSON(i).payload;
// Convert the ["0x81","0x25"...] to [0x81, 0x25]
var data = stringdata.map(function(item){return Number(item)});

It would be nice to have a way to get the octet sequence directly from the API to avoid the cast to and from JSON string, such as:

var data = input.samples.getByteArray(i,"payload");

[BrianAdams]

From a performance standpoint, I did a quick test comparing the two patterns. As you can see below (and not a surprise) serializing and deserializing JSON on even a moderate amount of data gets expensive if you are doing it a lot.

//@ 30Hz
// This is accessing one property of a much larger message
// Used 45-56% of a cpu core
var channel = Number(input.samples.getJSON(i).channelId);

vs

// User 2.2% of a cpu core
var channel = input.samples.getNumber(i,"channelId");

[gianpiero]

Hi @BrianAdams ,

With the next version of DDS the connector will return array of octets directly as values, so you won't have to do the mapping from string to value.

Right now you can get access directly the element of an array by using

input.samples.getNumber(i,"foo.myarray[5]");

or (if array of complex):

input.samples.getNumber(i,"foo.myarray[5].x");

I agree with you that a way to access the array as a whole could be useful, but as right now we don't have that capability.

[pwolf15]

@gianpiero and @BrianAdams my team would appreciate this feature as well. We are publishing and subscribing to several sets of image data (up to 640 by 480 @ 10Hz) stored as octet sequences. If possible, we would like to avoid the costs of serialization and deserialization. This API update would be ideal.

[BrianAdams]

@gianpiero glad to see 5.3.0.0 rolling out. Is the sequence support going to be part of this upgrade in the end?

[gianpiero]

@BrianAdams I am afraid is not. Thanks to yours an other user feedback, better support for sequences and array will be part of the list of feature we will consider for the next release. We are finishing up 5.3.0 (not just connector) these days. Then we will start discussion for next steps.

Thanks!

[spiderkeys]

Hi @gianpiero

Curious to know if there are any updates on the road map for efficient sequence access. This would unlock many use cases and capabilities for Connector, particularly the ability to create python tools that can interact with video streams.

[gianpiero]

Hello @spiderkeys,

Unfortunately I do not have any update on this.

Regards, Gianpiero

[jasontiller]

Any action on this? Still no implementation and it's almost a year later. We are actively researching using the Python connector and the abysmal performance of sequences is the major stumbling block.

[gianpiero]

Hello @jasontiller,

Unfortunately the Connector has been implemented as part of a research project that exhausted its funding: that's why you are not seeing many progress being made. The good news is that it will be supported in the next release of RTI Connext.