toBlob() to grab webcam images instead of toDataURL()

[alexanderkoumis]

This might be more of a sensor_msgs question but I'm wondering if the toBlob() function currently in FF and Chrome's Experimental Canvas Features can be used to grab the webcam frame instead of toDataURL(). As explained here toBlob() grabs the compressed jpeg or png's binary data without later expanding it to a base64 string, avoiding a bit of overhead. I'm curious to see if the 'sensor_msgs/CompressedImage' type used to publish an encoded image is meant only for base64-encoded data, or if it works with the binary files as well (will try soon and report performance impact).

[Rayman]

To what code are you refering? This example?

I think you are able to avoid the overhead you are talking about when you use BSON encoding for rosbridge, with that you can send raw uint8[] over websocket which you can obtain from the toBlob method. To enable this you have to do two things:

• for rosbridge_websocket, set the binary_encoder parameter to bson
• include bson.js to your html code (roslib will warn if you forget this)

[alexanderkoumis]

Hi @Rayman, thanks for your response. I'm sending over the binary object but am having trouble finding where the binary_encoder parameter is. I've grep'd around and the only mention of this parameter I can find is in /python2.7/dist-packages/rosbridge_library/internal/message_conversion.py. Also tried inserting it as a parameter in /opt/ros/jade/share/rosbridge_server/launch/rosbridge_websocket.launch and tried doing this in my launch file (feels very wrong):

<include file="$(find rosbridge_server)/launch/rosbridge_websocket.launch">
    <param name="binary_encoder" value="bson"/>
</include>

How can I set this parameter?

[Rayman]

I think this is all very experimental, because there is not much documentation about this stuff. I discovered this by reading the source.

I think you are supposed to set that parameter file in your launch files just like you did, see message_conversion.py. I'm also doing that on our robot: mobile_ui_server.launch.

If ROS encounters a uint8[] or char[] in a message, it will encode it using BSON instead of JSON. sensor_msgs/CompressedImage data field uint8[] so this should work. Note that base64 only increases the size by 37%.

I'm not sure if BSON has been implemented in two-way from rosbridge. Maybe someone else could answer this? rosbridge --> roslib (experimental) roslib --> rosbridge (???)

[alexanderkoumis]

Thanks for your example, still running into a few problems.

Here is the relevant bit from my launch file (copied from your example):

<!-- Launch the websocket server (communication via roslib) -->
<node name="rosbridge_websocket" pkg="rosbridge_server" type="rosbridge_websocket" output="screen">
    <param name="authenticate" value="false" />
    <param name="port" value="9090"/>
    <param name="address" value=""/>
    <param name="binary_encoder" value="bson"/>
</node>

With the image converted into a Uint8Array, as follows:

canvas.toBlob( function( blob ) {
    var reader = new FileReader();
    reader.addEventListener("loadend", function() {
        var byteArray = new Uint8Array( reader.result );
        var imageMessage = new ROSLIB.Message({
            format : 'jpeg',
            data : byteArray
        });
        topics['image'].publish( imageMessage );
    });
    reader.readAsArrayBuffer( blob );
}, 'image/jpeg');

rosbridge outputs the following errors:

[WARN] [WallTime: 1445460508.844563] Inbound TCP/IP connection failed: field data must be a list or tuple type
[ERROR] [WallTime: 1445460508.859205] [Client 5] [id: publish:/web/input/compressed:29] publish: field data must be a list or tuple type

I have made sure to source the bson.js file you provided. Worth note however I am able get the following to work, and from what I can tell it looks faster (haven't actually done bench-marking) due to the asynchronous image encoding, but the data is still in base64.

canvas.toBlob( function( blob ) {
    var reader = new FileReader();
    reader.addEventListener("loadend", function() {
        var imageMessage = new ROSLIB.Message({
            format : 'jpeg',
            data : reader.result.replace("data:image/jpeg;base64,", "");
        });
        topics['image'].publish( imageMessage );
    });
    reader.readAsDataURL( blob );
}, 'image/jpeg');