Image data is sent incorrectly to streetscape

[Algirdyz]

I am trying to stream image data to streetscape. I have a very simple implementation:

if(this.dataForCurrentFrame){
       var array = this.nodeBufferToTypedArray(this.dataForCurrentFrame);
       xvizBuilder
                .primitive("/camera/test_stream")
                .image(array, 'png')
                .dimensions(128, 128)
                .position([1,1,1]);
}

data in dataForCurrentFrame variable is binary buffer data received over a TCP socket. If I use this to save it, it works fine, I can open the image.

fs.writeFile('image.png', imagedata, 'binary', function(err){
            if (err) throw err;
            console.log('File saved.');
        });

When I look at frame data that is being sent to streetscape it looks like this:

"/camera/test_stream":{"images":[{"data":{"0":255,"1":216,"2":255,"3":224,"4":0,"5":16,"6":74,"7":70,"8":73,"9":70,"10":0,"11":1,"12":1,"13":1,"14":0,"15":72,"16":0,"17":72,"18":0,"19":0,"20":255,"21":254,"22":0,"23":19,"24":67,"25":114,"26":101,"27":97,"28":116,"29":101,"30":100,"31":32,"32":119,"33":105,"34":116,"35":104,"36":32,"37":71,"38":73,"39":77,"40":80,"41":255,"42":219,"43":0,"44":67,"45":0,"46":3,"47":2,"48":2,"49":3,"50":2,"51":2,"52":3,"53":3,"54":3,"55":3,"56":4,"57":3,"58":3,"59":4,"60":5,"61":8,"62":5,"63":5,"64":4,"65":4,"66":5,"67":10,"68":7,"69":7,"70":6,"71":8,"72":12,"73":10,"74":12,"75":12,"76":11,"77":10,"78":11,"79":11,"80":13,"81":14,"82":18,"83":16,"84":13,"85":14,"86":17,"87":14,"88":11,"89":11,"90":16,"91":22,"92":16,"93":17,"94":19,"95":20,"96":21,"97":21,"98":21,"99":12,"100":15,"101":23,"102":24,"103 ... etc.

but in streetscape i get an

Uncaught (in promise) DOMException

it is located in this file: https://github.com/uber/streetscape.gl/blob/master/modules/core/src/utils/image-buffer.js

when creating a new Blob(); The error says that data could not be decoded.

I am not sure where the actual error in this chain is. So I wasn't sure if should post the issue on streetscape repo or here. I decided to do it here because I do not like how that frame data looks like. It's an object, not an array, but I am pretty sure I send an array to xvizBuilder. If I try to print it out right before calling the builder it looks like this:

Uint8Array [
  255,
  216,
  255,
...
  21,
  21,
  21,
  12,
  ... 1816 more items ]

[Algirdyz]

So I made it work by first converting the frame to glb using the xviz writer and then sending it Now everything is fine.

Is sending images over json not supported?

[jlisee]

We'll have to look at this more closely. When sending the data as JSON it should be sent as a base encoded buffer not a binary array like that. So it should be:

"/camera/test_stream":{"images":[{"data":"TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5IGJ5IG...

So the things that are possibly broken here are:

• The encoder in pure JSON form (definitely looks broken)
• The decoder of the JSON data (possibly broken)

Right now we have only been doing this in glb for efficiency reasons but we should definitely support both.