plugin: ubuntu_build ZIP libPublisherVideoViewer.so

[ntegan1]

currently trying out hardware video decoding _media_player->setVideoDecoderPriority(QStringList() << "VAAPI" << "FFmpeg");

[ntegan1]

since qtav is discontinued (no libqtav-dev on ubuntu repos anymore) in favor of wang-bin's mdk-sdk,
I'm spending some time learning how the plotjuggler videoviewer plugin utilized openGL/software rendering via QtAV

it seems like we setup our own GL rendering window in openpilot/selfdrive/ui/qt/widgets/cameraview.h so I was going to see how that works and if any of that can be applied here.

tried compiling qtav master but there are some deprecated ffmpeg/libav audio APIs used that seem to be relatively easily replaced.

• QtAV/src/AudioResamplerTemplate.cpp replace swr_alloc_set_opts with swr_alloc_set_opts2
• QtAV/src/AudioResamplerTemplate.cpp replace av_get_default_channel_layout and av_get_channel_layout_nb_channels
• QtAV/src/AVCompat.cpp avcodec_find_decoder AVCodec* vs const AVCodec*

rerun is nice but doesn't support h265 with their rerun-mp4 create and doesn't support custom transformations/functions like plotjuggler does

==> I will be learning more about how the current PJ videoviewer plugin works and maybe submitting patches to QtAV to work with ubuntu 24's ffmpeg 7.
Otherwise will be replacing QtAV usage with raw ffmpeg libav frame reading and openGL image presentation inside PJ's videoviewer qt widget.

also considering using libmpv seems like straightforward c api
in openpilot/tools/plotjuggler/juggle.py we download to a temp rlog file, might as well download/concat hevc to a mp4 (for seekable timestamps in mpv) which would be easy to use with libmpv

// https://github.com/wang-bin/QtAV/blob/master/src/QtAV/VideoOutput.h
/*!
 * \brief The VideoOutput class
 * A VideoRenderer wrapper with QObject features. If create VideoOutput without a given renderer id, QtAV will try to create a widget based renderer, and dynamically load QtAVWidgets module if it's not loaded.
 */

    /*!
     * \brief VideoOutput
     * Create a QWidget based renderer. Result can be a QOpenGLWidget or QGLWidget based renderer if possible. Otherwise fallback to a software renderer
     */
    VideoOutput(QObject *parent = 0);

[ntegan1]

qtmultimedia5 videowdidget example loads h265 and h265 with a gstreamer plugin, i'll see how hard it is to change plotjuggler videplayer plugin to use this.

[ntegan1]

video_.zip

modified the four cpp source files for VideoViewer to replace qtav usage with qtmultimedia and qtmultimediawidgets
Had to install a gstreamer (libav or plugins-base) plugin on ubuntu 24.10 to get it to play h265.

This solution seeks and starts playback of the video to the current timeline time rather than the QtAV solution which decoded to individual frames via a FrameReader and stored them in a compressed format with qoi.h followed by presenting the frames via looking up frame number via a camera frame number time series from our logs.

plotjuggler's statepublisher API gives us two notifications of current time from mainwindow.cpp. In onplaybackloop or something, it calls it->second.play(absolute_time) or so after iterating the _state_publishers which calls our statepublisher videoviewer plugin with the current absolute time which seems to relate to our logmonotime which could be used to lookup a framenumber Y value by log time X value. I just use the first time given as the 0 time from qtmultimedia's perspective of the video file.
So PlotJuggler tells us when the video is playing (and the current timestamp many) many times per second.
It also tells us via updateState (again called in mainwindow.cpp) when the PJ time tracker was moved manually. I use this to pause the video and qtmultimedia mediaplayer setPosition to the correct time. A bug is that sometimes I have to drag the time slider back to 0 to get the correct absolute_time to subtract to convert PJ absolute_time, which correlates to our logmonotime, to a video-relative time for multimediaplayer.

May be useful to try libmpv which would allow for finer grained media control like perhaps seeking by frame or doing something like our framereaders in openpilot which use libav (https://github.com/leandromoreira/ffmpeg-libav-tutorial) to decode frames and compress them info memory like the QtAV plugin did

[ntegan1]

Screencast_20250123_133410.webm

my ryzen 5 5500u, 6gbram laptop struggles zooming in and out of a full 30 minute route on PJ
but still plays video pretty well, a bit choppy when screenrecording with KDE's spectacle utility