Last part gets discarded when terminating boundary is missing

[axit-joost]

trafficstars

TL;DR> The last part of a stream is discarded when a terminating boundary is missing. In the case when a Content-Length is known for the part (e.g. Motion JPEG streams over HTTP), we propose sending on the last part when the length of the part matches the Content-Length in the last boundary.

We are using dicer to transform a http multipart Motion JPEG stream into individual JPEG frames. During the creation of the testsuite for our 'MjpegReader' we created a sample. This will give our CI bot something to work on without requiring access to a camera. We created the sample.mjpeg using curl, e.g.

curl -m 1 http://somecamera/video.cgi > out.mjpg

The -m 1 switch 'cuts' the download after 1 second, which results in a captured mjpeg stream of 1 second and 15 frames.

Feeding this sample to dicer causes the last frame to drop. After investigating, we found that dicer is expecting an ending boundary, e.g.

--myboundary--

as per RFC 1341 (http://www.w3.org/Protocols/rfc1341/7_2_Multipart.html)

We notices that your testsuite has a testcase for this scenario, but that is limited to multipart form data for an uploaded file, which is a use case that is differing from our use case of streaming mjpeg data.

In our use case, we will be faced with connections dropping and therefore the terminating boundaries most likely always missing. We have attempted to listen to emitted events, but we were not able to coax dicer in to giving us that (potentially complete) data.

But since we know what the Content-Length is of the part in question, we (or perhaps even Dicer itself) can acertain that the part is complete and still send it on anyway.

--myboundary
Content-Type: image/jpeg
Content-Length: 68463

<partdata>

Insights, feedback and comments are very welcome.

[mscdex]

I'm not sure what the problem is. Once a part has started, data is just passed on through, so you should get any partial data. The only thing is that you won't see an end event on the part if the boundary is not seen.

Can you come up with a simple, reproducible test case?

[axit-joost]

I will see if I can cook up something simple.

It was really weird. We were feeding 15 frames in and getting 14 frames out. When using those 14 as input again, we got 13 frames out. Until we added a terminating boundary: then we got 15 frames in, 15 frames out. We were even able to reproduce a 100% byte accurate reproduction of the input stream again, so... there's definitely something strange going on there.

[amitguptagwl]

@axit-joost have your problem solved?

[axit-joost]

Not to my recollection. I did not have the time to cook up that reproducible test. In the mean time we pivoted our project and abandoned the original code base. Our current code does not depend on dicer, so this issue is now a moot point.