qgroundcontrol
qgroundcontrol copied to clipboard
mavlink
MacOS: update GStreamer 1.18.1 to 1.18.6 and fix CI (#10393 / #10408)
The CI builds on master for MacOS do not work. This PR updates the CI for MacOS and
- MacOS CI: Use official gstreamer package by picking patches from @patrickelectric from stable
- MacOS CI: Update gstreamer from 1.18.1 to 1.18.6 for MacOS
With GStreamer 1.18.6 the application does not crash anymore from the videofeed
gst-launch-1.0 videotestsrc ! x264enc ! rtph264pay ! udpsink host=127.0.0.1 port=5600
Closes: #10393 Closes: #10408
@patrickelectric : I rebased the patch and I did some further testing.
I have three testcases where I test the video display and the recording function.
Testcase UDP/I420
I generate the video stream with gstreamer with:
gst-launch-1.0 videotestsrc ! 'video/x-raw, width=640, height=480, format=I420' ! x264enc ! rtph264pay ! udpsink host=127.0.0.1 port =5600
The QGC Settings in "Application Settings -> General / Video Settings" are:
- Source: "UDP h.264 Video Stream"
- UDP Port: 5600
- Video decode priority: "default"
With this testcase the gstreamer pipeline uses the MacOS Videotools hardware decoder "vtdechw".
Testcase UDP/Y444
QGC Settings the same as UDP/I420. Only the format changed to Y444.
gst-launch-1.0 videotestsrc ! 'video/x-raw, width=640, height=480, format=Y444' ! x264enc ! rtph264pay ! udpsink host=127.0.0.1 port =5600
The Videotools hardware decoder cannot decode the Y444 format and therefore the avdec_h264 software decoder is used.
Testcase RTSP/Bunny The QGC Settings are:
- Source: "RTSP Video stream"
- RTSP URL: rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4
Testresults on MacOS
Testresults MacOS Testcase UDP/Y444 The video is displayed and the recording works. With VLC Player i can play the mkv file but mp4 and mov area played but black. With Apple Quicktime cannot play any of these files.
Testresults MacOS Testcase UDP/I420 When QGC ist launched and then the stream is started, the video display is either black or some flickering with gui elements is shown. The gui then looks like this:
https://user-images.githubusercontent.com/1280457/189139686-bbb1a586-777b-4d28-b1e8-be913ee3da11.mov
When the stream is then stopped and started again, then the normal video is shown. I think this is related to the use of the Apple Videotools hardware decoder. The problem is gone when I set the "Video Decode Priority" to "Force software decoder".
The recording works but only for the mkv file with VLC. The mov and mp4 are black in VLC. In Quicktime the mp4 and mov video is not changing, i.e. it is a still image.
Testcase MacOS RTSP/Bunny Sometimes the streaming starts and sometimes it does not start with messages like
VideoReceiverLog: Starting decoding "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4"
VideoReceiverLog: Stream timeout, no frames for 3 "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4"
VideoReceiverLog: Stopping "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4"
GStreamerAPILog: <source> receive interrupted
GStreamerAPILog: <source> receive interrupted
GStreamerAPILog: <source> PAUSE interrupted
GStreamerAPILog: <source> Timed out waiting for TEARDOWN to be processed.
GStreamerAPILog: <source> receive interrupted
GStreamerAPILog: <source> receive interrupted
GStreamerAPILog: <source> TEARDOWN interrupted
VideoReceiverLog: Streaming did not start "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4"
VideoReceiverLog: Stopped "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4"
VideoReceiverLog: Starting "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4" , buffer 0
VideoReceiverLog: Started "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4"
VideoReceiverLog: Starting decoding "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4"
VideoReceiverLog: Streaming started "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4"
VideoReceiverLog: Decoding started "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4"
VideoReceiverLog: _onNewDecoderPad "rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4"
VideoReceiverLog: Unable to link video sink
VideoReceiverLog: _addVideoSink() failed
I think this is related to the timeout setting of 3s of the receiver here. If I set the timeout to 10s, then I see the following in the log:
GStreamerAPILog: <source> warning: Could not receive any UDP packets for 5.0000 seconds, maybe your firewall is blocking it. Retrying using a tcp connection.
and the movie is displayed. So probably the timeout is too short for rtsp connections. With the longer receiver timeout of 10 seconds the movie is always displayed.
The recording results are the same as for Testcase UDP/I420.
Android
Testresults Android Testcase UDP/I420 The video display after start of the app shows the same flickering problem as MacOS. When the stream is stopped and then restarted, the video is shown normally. When I switch to "Force software decoder", then no flickering happens also after the start of the app. The recording for mkv works. I could play the mkv with android video player. For mov I did not have a player on the phone and the mp4 shows a still image.
Testresults Android Testcase UDP/Y444 Video shows immediately. Recording works but the android video player shows garbage for mkv and for mp4 the video player refuses to play the video.
Testresults Android Testcase RTSP/Bunny The timeout problem results in permanent teardown and restart of the receiver and I managed to get the flickering display but it can be displayed when it starts correctly. Recording for mp4 shows as still image and for mkv it works.
Linux tested on Debian 11 in a VM on MacOS.
Testresults Linux Testcase UDP/Y444 The video is shown. Recording and playing works for all formats mkv, mp4 and mov.
Testresults Linux Testcase UDP/I420 The video is shown. Recording and playing works for all formats mkv, mp4 and mov.
Testresults Linux Testcase RTSP/Bunny Cannot start the video. I guess due to the timeout problem.
So it is not perfect, but the recording works again and you do not crash on MacOS with a videostream immediately. The flickering problem shows that the robustness can be improved... But I think it is more stable with the patch than without.
I tested the Linux AppImage with commit 806cd0c8b9df484f2 and the RTPS Bunny Video now displays without problems. This solves maybe #9440.
Further testing shows that I can crash the QGC Application on MacOS with
gst-launch-1.0 videotestsrc ! 'video/x-raw, width=720, height=576, format=I420' ! x264enc ! mpegtsmux ! queue ! udpsink host=127.0.0.1 port=5600
when I set the QGC Video Settings to:
- Source: MPEG-TS (h.264) Video
- UDP Port: 5600
- Video decode priority: Default (=> will use the Video Tools HW )
and start the stream directly after the start of QGC. The crash log of MacOS shows that there is problem in the thread switching (dispatch):
Thread 1 Crashed:: Dispatch queue: org.freedesktop.gstreamer.glwindow
0 libsystem_platform.dylib 0x00007fff698bd929 _platform_memmove$VARIANT$Haswell + 41
1 libgstgl-1.0.0.dylib 0x000000010c6eb1a0 _gl_buffer_map + 320
2 libgstgl-1.0.0.dylib 0x000000010c6e398c _map_data_gl + 236
3 libgstgl-1.0.0.dylib 0x000000010c6ee990 gst_gl_context_thread_add + 336
4 libgstgl-1.0.0.dylib 0x000000010c6e37f0 _mem_map_full + 48
5 libgstreamer-1.0.0.dylib 0x000000010ce79a47 gst_memory_map + 103
6 libgstgl-1.0.0.dylib 0x000000010c6fa923 _gl_mem_map + 195
7 libgstgl-1.0.0.dylib 0x000000010c6e398c _map_data_gl + 236
8 libgstgl-1.0.0.dylib 0x000000010c6ee990 gst_gl_context_thread_add + 336
9 libgstgl-1.0.0.dylib 0x000000010c6e37f0 _mem_map_full + 48
10 libgstreamer-1.0.0.dylib 0x000000010ce79a47 gst_memory_map + 103
11 libgstgl-1.0.0.dylib 0x000000010c6e78c0 _do_convert + 1488
12 libgstgl-1.0.0.dylib 0x000000010c70efb8 _run_message_sync + 24
13 libgstgl-1.0.0.dylib 0x000000010c7121fc __gst_gl_window_cocoa_send_message_async_block_invoke + 44
14 libdispatch.dylib 0x00007fff6966d6c4 _dispatch_call_block_and_release + 12
15 libdispatch.dylib 0x00007fff6966e658 _dispatch_client_callout + 8
16 libdispatch.dylib 0x00007fff69673c44 _dispatch_lane_serial_drain + 597
17 libdispatch.dylib 0x00007fff696745d6 _dispatch_lane_invoke + 363
18 libdispatch.dylib 0x00007fff6967dc09 _dispatch_workloop_worker_thread + 596
19 libsystem_pthread.dylib 0x00007fff698c8a3d _pthread_wqthread + 290
20 libsystem_pthread.dylib 0x00007fff698c7b77 start_wqthread + 15
I guess the "flickering" behaviour is closely related to this. The "flickering" was also in Android: https://github.com/mavlink/qgroundcontrol/pull/10380#issuecomment-1237413448
Sometimes the video window shows some flickering before qgc finally crashes. When do the following:
- Start QGC
- Run Y444 stream:
gst-launch-1.0 videotestsrc ! 'video/x-raw, width=720, height=576, format=Y444' ! x264enc ! mpegtsmux ! queue ! udpsink host=127.0.0.1 port=5600=> Video is shown - Run I420 stream:
gst-launch-1.0 videotestsrc ! 'video/x-raw, width=720, height=576, format=I420' ! x264enc ! mpegtsmux ! queue ! udpsink host=127.0.0.1 port=5600=> Video is shown, no crash
This means that first running the pipeline with the software decoder and then using the hardware decoder make the pipeline work. I also managed to crash the app by using the UDP H264 stream with:
videotestsrc ! 'video/x-raw, width=720, height=576, format=I420' ! x264enc ! rtph264pay ! udpsink host=127.0.0.1 port =5600
That means that the video size compared to the previous test "Testcase UDP/I420" makes the difference between "flickering" and "(flickering then) crash".
The following receiver gstreamer receiver pipeline does not crash on command line:
gst-launch-1.0 udpsrc port=5600 caps='application/x-rtp' ! rtph264depay ! decodebin ! autovideosink
@andrewvoznytsa: I guess you designed the major rework of the video processing, e.g. bf928283d7873fd3e8. Do you have any idea where there may a thread communication problem which becomes visible only for the first pipeline attempt and with different latencies due to the hw decoding?
@andrewvoznytsa: Do I understand that correctly, that we have a qgroundcontrol gstqtplugin? https://github.com/mavlink/gst-plugins-good/compare/qgroundcontrol...GStreamer:gst-plugins-good:1.18
@fredowski yes, QGC uses statically linked qmlglsink for video rendering.
What I did in the past has been tested in YUV 4:2:0 case and worked on all platforms for all transports. H.265 and/or YUV 4:4:4 and/or something higher than baseline profile was never been priority here.
Unless you have really good reasons, I suggest to stick with one of YUV 4:2:0 variants (YV12, I420). IMO if you configure x264 to produce stream compliant with baseline/main profile then it will work as you expect.
Also it is not clear what you mean saying 'crash'. Do you mean some seg fault or did you want to say that the receiver pipeline is not able to decode the stream?
It is not clear what flickering issue is - do you mean flickering related to video rendering or do you mean stream reconnect and UI flickering?
@andrewvoznytsa: Thanks for your answer. To clarify:
crash means that the QGC application crashes, i.e. segfault. In the comment https://github.com/mavlink/qgroundcontrol/pull/10396#issuecomment-1241749622 I cited from the crash reporter of MacOS. The "flickering" refers to the display in the video window. You can see an example in the movie in the comment https://github.com/mavlink/qgroundcontrol/pull/10396#issuecomment-1240935400 or in the comment for Android https://github.com/mavlink/qgroundcontrol/pull/10380#issuecomment-1237413448.
The video formats that I chose are more or less arbitrary but the I420 is the problematic one here because that triggers the hardware decoder and results in the crash.
@andrewvoznytsa : Could you maybe give gst-launch examples to produce the video streams that you consider relevant?
@fredowski the flickering can be related to B frames (main/high profiles) and/or (very unusual but possible) P frame recording in baseline profile. I suggest to force baseline profile encoding and see if it disappears. Or disable B frames and then use whichever profile you need.
gst-inspect-1.0 x264enc will show you relevant options. Also you can use capsfilter after x264enc to force output format. But I suggest to use x264enc's options.
Regarding crash - sounds like it needs debugging. I suggest to dump pipeline graph and see caps in between decoder and qmlglsink. Maybe it can be also useful to examine pipeline and see what exactly is used to decode video. Perhaps it will give you some idea.
Sorry, I can't try to experiment right now. Maybe next week.
@patrickelectric: The gstreamer download from the freedesktop site takes over an hour in this run: https://github.com/mavlink/qgroundcontrol/actions/runs/3025642963 . Do you have access to the storage cloud and can you store the two 1.18.6 install files there as it is now?
@fredowski You may want to check out the repo here as they did some modifications to the gstreamer code for their android device. And from what I recall, NextVision has said that you must use baseline profile in QGC as well. You could also remove QGC's custom qmlglsink plugin and upgrade to the one straight from gstreamer based on the gstreamer version you are using, tho I don't know if it'll help any.
@patrickelectric : I have stripped this PR to have only the MacOS build working again including GStreamer. During the investigations I learned that not all Video formats are supposed to work. At least 1.18.6 does not crash for these formats. Now I tested on MacOS with
gst-launch-1.0 -vvvvv videotestsrc ! 'video/x-raw, width=1920, height=1080' ! x264enc ! 'video/x-h264, profile=baseline'! rtph264pay ! udpsink host=127.0.0.1 port =5600
i.e. with the standard FullHD Resolution in baseline profile. I tested the following resolutions leaving everything else the same.
- 1920 x 1080 => works
- 1280 x 720 => works
- 640 x 480 => works
- 320 x 240 => video remains in "Waiting for Video" - no video is shown.
Recording works and with the baseline profile also QuickTime can show the videos. I tested the videos also on Android and there also the 320 x 240 video works. However on Android the recording does not work (#9838).
@fredowski You may want to check out the repo here as they did some modifications to the gstreamer code for their android device. And from what I recall, NextVision has said that you must use baseline profile in QGC as well. You could also remove QGC's custom qmlglsink plugin and upgrade to the one straight from gstreamer based on the gstreamer version you are using, tho I don't know if it'll help any.
@holden-zenith : Thanks for the hints! It was really just the default example that failed and with standard resolutions and baseline profile life becomes much easier. I also tried the sinks from upstream (#10153) but it made no difference. So I just revert back to master with decodebin3 and the small 320x240 is simply not shown on MacOS (only with software decoder).
@fredowski You may want to check out the repo here as they did some modifications to the gstreamer code for their android device.
CubePilot is working on 4.0.8 that is before the major refactoring of the Videoreceiver code. I saw a modification of the latency parameter of the source but I do not understand what the purpose of that change is.
And from what I recall, NextVision has said that you must use baseline profile in QGC as well.
Yes, that makes it much easier and some other streams and resolutions do not work.
You could also remove QGC's custom qmlglsink plugin and upgrade to the one straight from gstreamer based on the gstreamer version you are using, tho I don't know if it'll help any.
As mentioned in the previous comment I tried that based on PR #10153 but it did not make a difference. I think the pipeline should also work with decodebin instead of decodebin3 and the "flickering" should never happen. My suspicion is that this is a bug in inter thread communication or usage of the videosink when it is not ready but I will not investigate this further.
Anyway I removed all changes of the videoreceiver code in this PR, except the update of GStreamer from 1.18.1 to 1.18.6 only for MacOS.
@patrickelectric hey did we ever figure out why did we leave this from main branch in the first place and only merged in 4.2 branch? is this good to go?
