How to send events and binary data in socket.io

[joelzavala]

Hi Links2004, I saw that your library includes support for communication with Socket.IO and I was wondering how to broadcast a simple event (with some data). This library is very good but I can't send binary data, so I would like to do it with your library. I hope your answer soon, my friend, thank you in advance

[Links2004]

the Socket.IO protocol is not designed to handel binary data, since all data is encoded in JSON. with this lib you can use plain websockets which supports binary data.

in the websocket "mode" you can use sendBIN via:

https://github.com/Links2004/arduinoWebSockets/blob/410489f7c5234690e5874e6731ed34cfe1305e9d/src/WebSocketsClient.h#L87-L88

and receive binary via the callback WStype_BIN:

https://github.com/Links2004/arduinoWebSockets/blob/410489f7c5234690e5874e6731ed34cfe1305e9d/examples/esp8266/WebSocketClient/WebSocketClient.ino#L41-L47

for big data steams take a look at the Fragmentation features of websocket connections.

https://github.com/Links2004/arduinoWebSockets/blob/410489f7c5234690e5874e6731ed34cfe1305e9d/examples/esp8266/WebSocketServerFragmentation/WebSocketServerFragmentation.ino#L46-L59

[joelzavala]

I understand my friend, another question, how can I broadcast data with an event with your library, for example, webSocket.emit("esp32test", "test"); // Event, Data

I've been reading this documentation and I see that it talks about binary events, I don't know if it has anything to do with what you're telling me

[Links2004]

are you talking about a websocket server on the ESP and want to broadcast to all clients? if yes look at:

https://github.com/Links2004/arduinoWebSockets/blob/410489f7c5234690e5874e6731ed34cfe1305e9d/src/WebSocketsServer.h#L71-L72

if we talking about a websocket client on the ESP, then your server code needs to handle that, since the client has only the connection to the server and does not know about any other connected clients the server may have.

what the documentation shows is how socketio encoded binary data, which is possible to recreate with this lib but socketio where never really designed for it, thats why its this hacky.

to send it like documented there you need to create the JSON and append your binary data to it. something like this:

// 51["project:delete",{"_placeholder":true,"num":0}] + <Buffer 01 02 03>

uint8_t buffer[52] = "[\"project:delete\",\"_placeholder\":true,\"num\":0}]";
buffer[49] = 0x01;
buffer[50] = 0x02;
buffer[51] = 0x03;

// Send event        
socketIO.send(sIOtype_BINARY_EVENT, &buffer[0], 52);

[joelzavala]

Great, now, what I have is a buffer of an image obtained (I'm working with an esp32-cam) and the image returns it to me in fb->buf and the size in fb->len

webSocket.loop();
if (millis() - last_time > 5000)
{
     last_time = millis();
     camera_fb_t *fb = NULL;
     fb = esp_camera_fb_get(); 
     webSocket.emit("esp32test", fb->buf, fb->len);
     esp_camera_fb_return(fb);
}

I would like to know where the image buffer would put it or how to know in which position each thing goes

[Links2004]

ok the buffer needs the size of JSON lengh + fb->len, since the JSON is dynamic too there is no fixed position to memcpy your data too. you need to use for example strlen() or .length() if its a Arduino String.

untested code example:

        // creat JSON message for Socket.IO (event)
        DynamicJsonDocument doc(128);
        JsonArray array = doc.to<JsonArray>();
  
        // add evnet name
        // Hint: socket.on('event_name', ....
        array.add("esp32test");

        // add payload (parameters) for the event
        JsonObject param1 = array.createNestedObject();
        param1["_placeholder"] = true;
        param1["num"] = 0;
        
          // JSON to String (serializion)
        String jsonoutput;
        serializeJson(doc, jsonoutput);
        size_t buf_size = jsonoutput.length() + fb->len;
        uint8_t *  buffer = new uint8_t[buf_size];
      
        memcpy(jsonoutput.c_str(), &buffer[0],  jsonoutput.length());
        
        memcpy(fb->buf, &buffer[jsonoutput.length()-1], fb->len);
      
        socketIO.send(sIOtype_BINARY_EVENT, &buffer[0], buf_size);
      
      

[joelzavala]

In the code part memcpy(jsonoutput.c_str(), &buffer[0], jsonoutput.length()); the compilier says me argument of type "const char *" is incompatible with parameter of type "void *"

I replaced it by memcpy(const_cast<char *>(jsonOutput.c_str()), &buffer[0], jsonOutput.length()); and compiled Every time the esp32 sends the package, the server shows me this error

Missing error handler on socket. Error: Illegal attachments at decodeString (C:\Users\chiki\Desktop\CamaraSeguridad\node_modules\socket.io-parser\index.js:293:13) at Decoder.add (C:\Users\chiki\Desktop\CamaraSeguridad\node_modules\socket.io-parser\index.js:242:14) at Client.ondata (C:\Users\chiki\Desktop\CamaraSeguridad\node_modules\socket.io\lib\client.js:195:18) at Socket.emit (events.js:223:5) at Socket.onPacket (C:\Users\chiki\Desktop\CamaraSeguridad\node_modules\engine.io\lib\socket.js:109:14) at WebSocket.emit (events.js:223:5) at WebSocket.Transport.onPacket (C:\Users\chiki\Desktop\CamaraSeguridad\node_modules\engine.io\lib\transport.js:105:8) at WebSocket.Transport.onData (C:\Users\chiki\Desktop\CamaraSeguridad\node_modules\engine.io\lib\transport.js:116:8) at WebSocket.onData (C:\Users\chiki\Desktop\CamaraSeguridad\node_modules\engine.io\lib\transports\websocket.js:79:30) at WebSocket.emit (events.js:223:5)

I think this is already another issue but any help is good for me

[Links2004]

hard to say its possible the format is not correct yet. may send some binary data via the nodejs client / server and compare how the data that is send is different (F12 in the browser helps there a lot).

like I sad the code I posted is untested code example, so its easy possible that the format is not 100% what socket.io expects.

[joelzavala]

I was researching and I was misplacing the parameters in the memcyp function, the specification says that the first parameter is the destination, the second is the source and the third is the length of the source. In this case it would be like this

memcpy(&buffer[0], jsonoutput.c_str(), jsonoutput.length());

the same would be with the image buffer

memcpy(&buffer[jsonoutput.length()], fb->buf, fb->len);

I think that's why the server told me that an error ocurred decoded the string

[joelzavala]

in the client i execute this

const test = new Uint8Array(5)
test[0] = 44
test[1] = 45
test[2] = 46
test[3] = 47
test[4] = 48

socket.emit("message", test)

and this is what the socket sends

Aparenttly, the socket separate the string and the binary is encoded to base64, what do yo think about it?

[Links2004]

you are looking at the polling mode here, this is no websocket ;) you need transport=websocket in the URL

[joelzavala]

I´m tried to send the data but the browser doesn´t send, this is what i send

[Inglebard]

Hi,

It seems I try to do something similar. A part of my code is available here (base64 version) : https://stackoverflow.com/questions/66549640/esp32-cam-fastest-way-to-stream-video-output-to-nodejs-server-with-socketio I now try to convert the base64 version to binary version because of performance issue.

Did you succeed to send binary data via socketIO ?

[joelzavala]

according to the socket.io documentation it say that send 51-["hello",{"_placeholder":true,"num":0}] + base 64 encoded binarybuffer

[Inglebard]

I always use json data in socketIO. I don't know much about binary data in socketIO and websockets. The examples present in this issue doesn't seems to work for me. For example the simplest example present here (https://github.com/Links2004/arduinoWebSockets/issues/603#issuecomment-758163316) from @Links2004 trigger SocketIO error "Illegal attachments".

[joelzavala]

If you enable debug option in arduino, you will see that when it recive a binary event ( when server send a binary data ) it receives it as I told you.

just you have to convert your binary image to base64 and paste it after the json data and emit with binaryevent(string)

[Inglebard]

@joelzavala So I don't understand :laughing:. Sorry if I'm Off topic.

If you need to base64 the buffer, it's not "real binary" data.

Also If it is required to base64 your binary data why use binary instead of text (JSON). Because If I compare the workflow : Text(JSON) : data -> base64 binary (img for example) -> convert to JSON -> send text event Binary : data ->convert to JSON -> then convert to uint8_t ->base64 binary (img for example) -> merge all data -> send binary event

So binary event require more "transformations" than text event, and they require both base64 and JSON so I don't get why use binary.

I expected socketIO binary event to be similar to websocket and the possibility to send a binary data directly like this webSocket.sendBIN(fb->buf,fb->len); (not checked received data but send/receive works).

EDIT: I switch to websocket to use binary data, project here : https://github.com/Inglebard/esp32cam-relay

[joelzavala]

Is that socket.io is not intended for binary data transmission and the addition of this functionality is a plus.

[Inglebard]

Sorry to insist, but just to be sure. My error is "Illegal attachments", if I check socketIO source code : https://github.com/socketio/socket.io-parser/blob/af1b23ca85fb64a0d7a050abd0362c5f632ce429/lib/index.ts#L191

It is trigger before checking the data. It is caused by encoding format. The encoding format, as @joelzavala notice, must be : <packet type>[<# of binary attachments>-][<namespace>,][<acknowledgment id>][JSON-stringified payload without binary]+ binary attachments extracted, ex:51-["hello",{"_placeholder":true,"num":0}] + <Buffer 01 02 03>.

If I take this example :

// 51["project:delete",{"_placeholder":true,"num":0}] + <Buffer 01 02 03>

uint8_t buffer[52] = "[\"project:delete\",\"_placeholder\":true,\"num\":0}]";
buffer[49] = 0x01;
buffer[50] = 0x02;
buffer[51] = 0x03;

// Send event        
socketIO.send(sIOtype_BINARY_EVENT, &buffer[0], 52);

I can identify all parameter except "the number of binary attachments". "the number of binary attachments" even if no binary attachment, 0 must be set.

It seems to miss a function call : 'socketIO.sendBINEVENT(numberofbinattachement, &buffer[0], 52);' => with implicit sIOtype_BINARY_EVENT type which need to concat sIOtype_BINARY_EVENT + numberofbinattachement. Since this library do not entirely implement socketIO binary event, can @Links2004 confirm this is the issue or am I completely wrong ?

[joelzavala]

I have been investigating more and it seems that it does two websocket sends, one where it says the structure and another where it sends the pure binary data.

In this example, send binary data as base64

[Inglebard]

Hi, here is a working example on sending binary data without base64. Send binary event example

This example is based on this stackoverflow answer.