Synchronized Video Streaming with Raspberry PI Zero 2 W

[eliabruni]

Hello,

Great project! I would like to use this library for a specific use case, but I am not sure if it would be suitable so I'll ask here.

The use case involves using two Raspberry PI Zero 2 W devices to stream video feeds in parallel to a Linux terminal. The critical requirement is to align each video frame from one stream with the corresponding frame from the other stream, based on their timestamps.

To achieve this, the following functionalities are necessary:

Each video frame must be accompanied by a unique timestamp when sent from the Raspberry PIs. The receiving terminal should be capable of processing the streams by fetching frames individually, retrieving their timestamps, and synchronizing them with the frames from the alternate stream. Could you please advise whether your library supports these features, or if there is a possibility to implement such a synchronization mechanism using your library?

Thanks in advance!

[TzuHuanTai]

I record timestamps while receiving the frame from the camera, https://github.com/TzuHuanTai/RaspberryPi_WebRTC/blob/df98d7bbfcf4a44eb5ccfea360f04c117f66380f/src/track/v4l2_track_source.cpp#L76-L80

This timestamp is appended to the encoded frame before sending it to the webrtc client side. https://github.com/TzuHuanTai/RaspberryPi_WebRTC/blob/df98d7bbfcf4a44eb5ccfea360f04c117f66380f/src/v4l2_codecs/raw_buffer_encoder.cpp#L82

So It may work for your case😕. But you must ensure the clocks from different devices are synchronized, or there is always a fixed time difference between each device.

There are still a lot of codes that need to be cleaned or refactored in the future. I don't recommend using this project in production so far. I'm refactoring the DMA mechanism to enhance the performance in those few days, I plan to migrate this project to my pi zero 2 as well😄.

[eliabruni]

Ah, that's great! I'll try out the timestamp code then :) As for time sync, I can use ntp sync, shouldn't be a problem.

As for production warning, message received. I already like how well the code is structured, looking forward to seeing the further progress! If you need some help in testing, let me know, especially with pi zero 2!

[eliabruni]

Btw, for testing on pi zero 2, would you recommend installing RaspberryPi OS 64bit?

[TzuHuanTai]

Yes, I'm using Raspberry Pi OS 64bit. I haven't tested on others yet.😅

[eliabruni]

Alright, will try that (Bullseye 64bit)!

One more question: do build webrtc directly on RPI or do you cross-compile? I am asking because it might take a long time on pizero and am not even sure if it can do that. I am trying to figure out a good cross-compilation recipe..

[TzuHuanTai]

I built on my WSL with Ubuntu 20.04, and it only took 10 minutes. The compiling arguments are here.

[eliabruni]

Thanks. I have trying on bare ubuntun 22.04, but could also give WSL on 20.04 a try. Besides the compiling arguments you pointed at, you must also have a cross compile configuration/procedure right? I have been trying to cook one, but I am getting clang errors (despite using your recommended one). I am getting weird CLANG errors, e.g., there are several compiler flags that the current version of clang does not recognize (that's both with your optional procedures or the default vresion)

-Wno-thread-safety-reference-return -Wno-delayed-template-parsing-in-cxx20 -split-threshold-for-reg-with-hint=0

[TzuHuanTai]

Try to download the newer clang version here. It usually happens because the webrtc team uses the relatively new clang. It would show unrecognized errors If your clang is too old.

[TzuHuanTai]

I don't know whether you check out the specific branch of webrtc or the newest one. I'm using m115 now.

[eliabruni]

That worked, thank you! I am cross-compiling on ubuntu 22.04. Later on, if you want I can share the whole recipe from creating the cross-compilation env onward, maybe it could be useful info for this repo.

Now I am moving to microsoft-signalr.so. Did you also cross-compile that on WSL or did you instead directly compiled and install it on RPI?

[eliabruni]

In the meantime I made progress with cross-compilation, however I now get stuck at Install [cpprestsdk](https://github.com/Microsoft/cpprestsdk/wiki/How-to-build-for-Linux).

all works (and I followed carefully your instructions about c++11 --> c++14) until when i try to run

ninja

at that point i have a ton of compilation errors; i create a gist with the output

[TzuHuanTai]

I built all other libs on Raspberry Pi. webrtc.lib can be compiled on WSL because it provides a cross-compile arg target_cpu="arm64". We need to make sure all libs are compatible with arm based platforms.

[eliabruni]

Alright, will do the same then and let you know!

[eliabruni]

Compiling on pizero seems to be quite hard though. I am now at casablanca but in 8 hours it is at this step:

[5/160] Building CXX object Release/src/CMakeFiles/cpprest.dir/json/json.cpp.o

while using 2GB+ swap.

I wonder if we could compile on more powerful RPI and then transfer the library - would that work you think?

[TzuHuanTai]

I'm researching how to emulate Raspberry Pi OS on Windows to speed up. I plan to use QEMU to run an arm-based device with it. But I got stuck on the below command. I'm trying to figure it out. :(

qemu-system-aarch64 -machine raspi3b -cpu cortex-a72 -dtb "D:\bcm2710-rpi-3-b-plus.dtb" -kernel "D:\kernel8.img" -drive file="D:\2023-10-10-raspios-bookworm-arm64-full.img",if=none,id=drive0 -append "rw earlyprintk loglevel=8 console=ttyAMA0,115200 dwc_otg.lpm_enable=0 root=/dev/mmcblk0p2 rootdelay=1" -serial stdio -usb -device usb-mouse -device usb-kbd -device usb-net,netdev=net0 -netdev user,id=net0,hostfwd=tcp::5555-:22

[TzuHuanTai]

btw, my libs were built on 3B, the pi zero is too slow. I did transfer compiled libs to pi zero. I already tested the project on my pi zero two days ago, and it worked fine.

[eliabruni]

excellent, i will do the same then (suing RPI3 and transfer); as for the QUEMU aproach, let me open another issue for you with my current configuration on ubuntu 22.04 (but should work on 20.04 as well); maybe we can work on the env together

[TzuHuanTai]

I recompiled all dependent libs and released a static executable file here. You could take a try.

[eliabruni]

That's very kind, thank you (I deleted the message because right after I asked that the RPI4 arrived :)

[eliabruni]

I just tested your binaries and everything is working, thank you!

Back to the original question of this issue, for our purpose, we won't need a browser on the receiving end. We will instead have another arm64 linux machine, which will match images based on their timestamps and then send that to a neural network. So, I am now wondering if we should replace signalr with a different (simpler) communication protocol. In particular, because it probably doesn't make much sense to use signalr server hub, right? Despite that, signalr seems like a very nice library and I wonder if I could easily integrate in my usecase, with no browser.

[TzuHuanTai]

It's a good question! I agree with your suggestion that the signaling mechanism could be replaced or selected by other protocols. I'll try to decouple and refactor it with an interface in a few days. For the time to implement the regular web socket or other protocols, I have no plan for when I'll finish it. Maybe you can implement your protocol on the interface, and recompile the program by setting customized options like -DUSE_SIGNALR=true, -DUSE_WEBSOCKET=true or so. I initially chose signalr as a signaling mechanism because I have a project that broadcasts sensor data through signalr. Built on top of it is the most convenient way for me, that's all.

[eliabruni]

Having an interface would be great! Alright, I'll look into the best suitable signaling protocol for my use case while waiting for the interface then. I'll be happy to act as tester afterwards, maybe that could also be helpful for others if we document it well.

[TzuHuanTai]

I refactored the signaling service code in #86. For example, If you'd like to use websocket, the new command to set the signaling service would be "cmake .. -DCMAKE_CXX_COMPILER=/usr/bin/clang++ -DUSE_SIGNALING=WEBSOCKET", the default is SIGNALR. You can implement any protocols by inheriting the class in src/signaling/signaling_service.h, and creating the service in src/main.cpp like below.
https://github.com/TzuHuanTai/RaspberryPi_WebRTC/blob/01465be0cec3eaf09784c3b10a013a568cd526d5/src/main.cpp#L26-L34

[eliabruni]

That's great! I will give it a try and ask for feedback as I am not an expert :)

[eliabruni]

Quick question: which version of ffmpeg are you using on the RPI and was tested with?

[TzuHuanTai]

I use 4.3.6 version of ffmpeg, the shared libs details show with the command apt list --installed | grep libav

libavc1394-0/oldstable,now 0.5.4-5 arm64 [installed,automatic]
libavcodec-dev/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed]
libavcodec58/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavdevice-dev/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed]
libavdevice58/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavfilter-dev/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavfilter7/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavformat-dev/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed]
libavformat58/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavresample4/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]
libavutil-dev/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed]
libavutil56/oldstable,now 8:4.3.6-0+deb11u1+rpt5 arm64 [installed,automatic]

[eliabruni]

Thank you!

The receiver I am working on is a rock 5A,

Here my output of apt list --installed | grep libav

libavahi-client3/jammy-security,jammy-updates,now 0.8-5ubuntu5.2 arm64 [installed,automatic]
libavahi-common-data/jammy-security,jammy-updates,now 0.8-5ubuntu5.2 arm64 [installed,automatic]
libavahi-common3/jammy-security,jammy-updates,now 0.8-5ubuntu5.2 arm64 [installed,automatic]
libavahi-core7/jammy-security,jammy-updates,now 0.8-5ubuntu5.2 arm64 [installed,automatic]
libavc1394-0/jammy,now 0.5.4-5build2 arm64 [installed,automatic]
libavcodec-dev/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed]
libavcodec58/jammy,now 7:4.4.2-0ubuntu0.22.04.1+rkmpp20230327 arm64 [installed,automatic]
libavcodec60/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed,automatic]
libavdevice-dev/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed]
libavdevice60/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed,automatic]
libavfilter-dev/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed]
libavfilter9/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed,automatic]
libavformat-dev/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed]
libavformat60/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed,automatic]
libavutil-dev/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed]
libavutil56/jammy,now 7:4.4.2-0ubuntu0.22.04.1+rkmpp20230327 arm64 [installed,automatic]
libavutil58/jammy,now 7:6.0-5+git230804.e243e8d001~j1 arm64 [installed,automatic]

I also need to build a specific ffmpeg version.

Do you think this could cause issues? If I understand correctly, the thing the receiver should do is properly decode and scale the video stream. As far as I can see, these are the dynamic libs I should pay attention to:

set(WEBRTC_LINK_LIBS X11 dl)
set(FFMPEG_LINK_LIBS avformat avcodec avutil swscale avdevice)

[TzuHuanTai]

It'll be ok. The receiver only needs to decode the video stream. The different versions of ffmpeg may change the way to call their API, but codec standards are usually the same.

[eliabruni]

Glad to hear. When attempting to compile to project on rock5, I indeed get issues such as

RaspberryPi_WebRTC/src/common/recorder.cpp:80:14: error: cannot initialize a variable of type 'AVCodec *' with an rvalue of type 'const AVCodec *'
    AVCodec *codec = avcodec_find_encoder_by_name(encoder_name.c_str());
             ^       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

which I could fix by replacing

AVCodec *codec = avcodec_find_encoder_by_name(encoder_name.c_str());

with

const AVCodec *codec = avcodec_find_encoder_by_name(encoder_name.c_str());

I am working on the alternative signaling and am also writing the receiver side (i.e., I am writing a conductor.h for the receiver as well as its server; for the server, I am trying mosquitto).

[TzuHuanTai]

In this project, the ffmpeg is only used for recording the video file that I'm doing in issue #91 so far.