phoboslab
Stream corruption with separate audio and video inputs
So I've observed a significant latency when using one ffmpeg process to feed both video (from v4l2) and audio (from ALSA) to the included nodejs relay. With latest master. So I tried switching to having two ffmpeg processes: one for video and one for audio, as mentioned in the docs. First thing I got was the Google Chrome tabs crashing a random time after stream startup (1-4 seconds). I worked around that by disabling WebAssembly. That left the stream running in a stable way with audio working and low latency, but I was getting the message about skipping garbade data every few seconds. At that point also there was a single frame of partial corruption in the video as well - a few lines in the frame would get some corruption. Interestingly that only happened when audio was silent, there was no corruption while audio was playing music. Could it be that the silent audio is causing ffmpeg to create audio packets that confuse jsmpeg enough to try to interpret them as video data packets? That could explain quite a bit of problems people have been having.
On my side I've worked around this whole thing by having two separate ffmpegs, two relays and two canvas elements (one of them with display: none) for video and audio fully separate.
(Also stopping player with destroy() does not quite work safely with WebGL active, so I had to just make multiple players on the same canvas, pause() all and then play() the one I want to have active. The downside being that all streams are active and consuming bandwidth, but the upside is that switching is nearly instant :))
I can try to provide a Dockerfile to demonstrate the corruption issue if you can not reproduce it locally.
P.S. Great cudos and thanks for this library - it is a core part of a very important and popular internal project of mine! If you're ever around Munich, I owe you a beer!
I know this is tangential, but when you tried only 1 ffmpeg process did you have the audio sample rate set to 44100? I discovered the high latency went away for me if I set the sample rate to the maximum my audio capture device supported (48000 in my case).
I suspect that feeding 2 sources into the websocket is not as robust as claimed in the documentation, although I didn't see corruption when I tried myself. I plan on testing a bit more as I like this library and would love to get help make it more robust! That said, I may also have found a bug in the jsmpeg code that might trigger the error you are seeing (it looks for synchronisation bytes in the first 187 bytes instead of first 188 which means it might miss it sometimes!)
I stumbled across this discussion because i'm having dificulties in enabling JSMPEG audio on my iPhone with iOS 14.2. Because i wanted to use the microphone on my phone to be able to talk back listening to a live stream, i had to switch from http/ws to https/wss. With http jsmpeg allways worked flawlesly with two seperate input streams (audio and video). The switch to https introduced some additional latency and two seperate input streams even no longer worked. The video got distorted in a bad way, resembling what you experienced. When i switched to a single ffmpeg command it al worked, but after some time audio is sent over faster than the video and latency is noticeable.
I have changed my ffmpeg command to set a higher sample rate, from 44100 to 48000, just to be sure.
My setup: RPi 3B+ with Buster (planning to go for a RPi4 to see if that gives some performance improvement) NodeJS Node Red Lighttpd web server JSMPEG Jabra Speak 510 USB
The single command i use: ffmpeg -f v4l2 -framerate 25 -video_size 640x480 -i /dev/video0 -f alsa -ar 48000 -i plughw:CARD=USB,DEV=0 -f mpegts -codec:v mpeg1video -s 640x480 -b:v 2000k -bf 0 -codec:a mp2 -b:a 128k -muxdelay 0.001 https://localhost:8081/supersecret
BR.
Ed.
had to switch from http/ws to https/wss. With http jsmpeg allways worked flawlesly with two seperate input streams (audio and video). The switch to https introduced some additional latency and two seperate input streams even no longer worked.
A bit off-topic, but could you elaborate about the additional latency? How much it was? Did you notice other problems with SSL and jsmpeg?
I'm new to jsmpeg and need to use it with SSL and I was hoping for low latency without any measurable difference to non-SSL usage.
