rosbridge/JSON point cloud serialization performance

[AndreiBarsan]

I am playing around with webviz for visualizing a real-time application, so streaming data into it via rosbridge.

It seems that when webviz receives messages via rosbridge, things like the bytes which form a point cloud's data have to be encoded as a list of bytes (!!) in JSON, which is extremely inefficient.

Presumably just reading rosbags is fine, since they encode the bytes in a binary format, but encoding the bytes of a point cloud tensor as ASCII JSON is inefficient. A back of the envelope calculation shows that this representation is ~10x less efficient than a binary one :(

Is there a way to encode point clouds more efficiently when talking to webviz via JSON?

(I apologize if this is a silly question; I am not very familiar with ROS and its ecosystem.)

Thank you!!

[AndreiBarsan]

Note: It seems base64 is supported!

    l_bytes = base64.b64encode(lidar_xyz_bytes)
    lidar_b64 = l_bytes.decode("ascii")

and then putting the lidar_b64 string in the JSON I am sending with roslibpy to webviz via rosbridge seems ~6-7x faster than the ugly python-list-of-bytes I was originally using.

Is this how people typically encode messages, or are there even better ways?

[jhurliman]

Does your use case allow for connecting directly to a ROS node? If so, Foxglove Studio has native ROS1 support using the binary TCPROS protocol (https://github.com/foxglove/studio/pull/328).

[flynneva]

good suggestion with trying out foxglove, there might be some benefits to using that instead.

for webviz i think you can try to use the compression they support, i remember awhile back seeing a bunch of work done for cbor-raw compression for specifically this use case (raw binary data messages)