Order not preserved while sending data

[aniket-banyal]

This is a part of my code that I am using to transfer files.

 const CHUNK_SIZE = 16 * 1024
 file = e.target.files[0]
 let i
 let idx = 0
 for (i = 0; i < file.size; i += CHUNK_SIZE) {
     connection.send({ chunk: file.slice(i, i + CHUNK_SIZE), idx })
     idx++
 }

When this data is received on the other user's end then I print the idx received and all of them are in the right order except the last two. Example - 0 1 2 . . . 207 209 208

This is happening for all the files but most of the files are opening without any problem while some of them are not.

I tried setting the ordered attribute to true but it still remained false -

 connection = peer.connect(userId)
 connection.dataChannel.ordered = true

[Swanand01]

The same has been happening with me. The second to last chunk is always received before the last one.

[cohenstyle]

Yeah. It's not setting the DC to ordered unfortunately. Having the same issue.

[jonasgloning]

It's a bit more complicated: The problem is not ordering incoming data but outgoing data.

The binarypack encoding produces javascript Blobs, while the underlying data channel accepts only ArrayBuffers. Converting them is possible, but it is an async operation. Also, large objects must be sent in chunks (these are packed twice), so converting them takes even more time. Some small objects may reach the sending buffer before the encoding of the previous message is done.

The current system predates cheap ordered data channels, so extra care was taken to avoid making assumptions about the arriving order. That's also why the ordered attribute is not set automatically. The current solution was reasonable a few years ago.

We have a MsgPack/CBOR serialization (https://github.com/peers/peerjs/pull/1031) in the pipeline that doesn't exhibit this behaviour, uses reliable channels by default and is much faster than the current implementation. It works well, but there are still some things to figure out regarding backward compatibility and a possible plug-in system.

[github-actions[bot]]

:tada: This issue has been resolved in version 1.4.8-rc.1 :tada:

The release is available on:

• npm package (@rc dist-tag)
• GitHub release

Your semantic-release bot :package::rocket:

[jonasgloning]

I published a new beta release that should respect the ordering (except for File/Blobs) Please test it!

You can get it on npm via

npm i peerjs@rc

[cohenstyle]

Thanks a lot! I was using it with files though so I don't have a setup that can test it.

[jonasgloning]

Please don’t feel obligated to try, but maybe you could call file.arrayBuffer() before passing it to PeerJS. Then there’s no async operation left in the send operation, and the order can be guaranteed.

const file : File = ....
const ab : ArrayBuffer = await file.arrayBuffer();
dataConnection.send(ab)

We could also make send itself async, but awaiting this could be confusing as a return would only mean "in buffer", not "on the wire".

[brianismeta]

I was having an ordering issue but I thought it had to do with UDP... there are many data packets being transferred between two peers, and sometimes it gets out of order very quickly, or it gets out of order after a few hundred packets. I forced communication through TURN and used TCP channel, and the issue was resolved, as far as I could tell. I am assuming it was still a UDP issue but after reading this issue - is it possible it could have been this issue -- or does this issue only occur at the very "end" of the data channel communications (??) (Also my presumption is that UDP packets can get dropped anywhere along the way, whereas TCP will ensure it gets to the other side)

[jonasgloning]

Hi @brianismeta, WebRTC’s data channels use SCTP, not UDP or TCP. If you want to keep the order of messages, you’ll have to set {reliable: true} in the Peer.connect call. This setting will become the default soon. (Do you mean this with TCP channel?) Forcing the traffic over TURN has no influence on the order.

If your issue gets resolved when using a "TCP channel", you are probably not affected by this issue. If not, please give the beta version a try.

[github-actions[bot]]

:tada: This issue has been resolved in version 1.5.0-rc.1 :tada:

The release is available on:

• npm package (@rc dist-tag)
• GitHub release

Your semantic-release bot :package::rocket: