Weylus
Weylus copied to clipboard
H-M-H
Hardware Acceleration for RPi4 under 64-bit OS
Hey! first of all, thank you for developing this wonderful tool, it has so many purposes, and is portable to various platforms at the same time.
I decided to push it a little bit further, tho.. I built from source and installed the app in my RPi4 8G running under Pop!_Os 21 for arm64, and it works flawlessly. Using the full KMS, i get around a stable 15fps after some seconds upon connecting.
I've been trying to push it to at least 30 though, so I tried fixing the VAAPI binding, creating a symlink from /usr/lib/aarch64.../dri to /usr/lib/dri, and that seemed to make it load a bit further, but still no luck. after some research, I found that VAAPI isn't fully compatible with 64-bit arm, from what I understand..
So my question is: Is it possible to implement v4l2 someway to enable it to Weylus? maybe under ffmpeg?
Here's the error output:
Thanks!
This is interesting, it always makes me happy seeing all the different applications people find for Weylus!
Is it possible to implement v4l2 someway to enable it to Weylus? maybe under ffmpeg?
Yeah, that should be possible. Anything ffmpeg can encode, Weylus can also be made to encode :). But first try to check if ffmpeg itself works fine with v4l2, the following command closely resembles what Weylus needs to do internally:
ffmpeg -f lavfi -i testsrc2=size=1920x1080 -vcodec h264_v4l2m2m -t 10 -bf 0 -movflags frag_keyframe+empty_moov+default_base_moof test.mp4
I can't test this, so some tweaks may be required but once you get it working it's probably no different from what is done for libx264:
https://github.com/H-M-H/Weylus/blob/67500d87769c9b9c1acc87ebd1c15488d0c7a3d2/lib/encode_video.c#L359-L383
As quick and dirty hack you could just adapt above code to work with h264_v4l2m2m.
Wow! nice! I will try that, and conclude it here
I tried running the above said command, and it seemed to work! although the video was a black screen (i think that was because of missing codecs), which indicated that it found the right video encoder, and it seemed pretty fast when I ran compared with the default one!
I then tried to recompile the whole package with that portion you mentioned altered, but when connecting it says that the codec was not found.. I'll dig a bit deeper on another day, as it takes nearly 20 min to compile everything on my RPi, but it seems promising
Hi! I tried to compile h264_v4l2m2m into weylus as suggested by H-M-H. Weylus compiled without errors and video encoding is starting (see log below). Actually something goes wrong and the browser can't decode the video.
Any thoughts on this?
hule-ka:/usr/local/src/Weylus/target/release $ ./weylus --access-code=123 --auto-start --no-gui
2022-01-21T18:46:40.241783Z INFO weylus::web: Webserver listening at 0.0.0.0:1701...
2022-01-21T18:46:43.285481Z INFO weylus::websocket: Client connected. address=192.168.6.3:54286
2022-01-21T18:46:43.303811Z INFO weylus::websocket: WS-Client authenticated! address=192.168.6.3:54286
2022-01-21T18:46:44.115261Z INFO weylus::log: Using device /dev/video11
2022-01-21T18:46:44.115423Z INFO weylus::log: driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
2022-01-21T18:46:44.115524Z INFO weylus::log: requesting formats: output=YU12 capture=H264
2022-01-21T18:46:44.165597Z WARN weylus::log: Failed to set timeperframe
2022-01-21T18:46:44.166003Z WARN weylus::log: Failed to set gop size: Invalid argument
2022-01-21T18:46:44.178114Z INFO weylus::log: Video: 1920x1080@h264_v4l2m2m pix_fmt: yuv420p
2022-01-21T18:46:44.485334Z INFO weylus::log: ff_v4l2_buffer_buf_to_avpkt
2022-01-21T18:46:44.485408Z WARN weylus::log: Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2022-01-21T18:46:44.485442Z WARN weylus::log: Encoder did not produce proper pts, making some up.
2022-01-21T18:46:44.485760Z INFO weylus::log: ff_v4l2_buffer_buf_to_avpkt
2022-01-21T18:46:44.644296Z INFO weylus::log: ff_v4l2_buffer_buf_to_avpkt
Is the video playable at all (e.g. via ffplay or mpv)? It looks like h264_v4l2m2m doesn't support all the options Weylus uses. You might want to play around with these parameters: https://github.com/H-M-H/Weylus/blob/33cf5f138b5a7eefa8cf07c71eb0403a4fce849f/lib/encode_video.c#L121-L126
It's also possible that h264_v4l2m2m doesn't like encoding at a variable framerate. Weylus sets a so called time base to 1/1000, which means timestamps of frames are recorded as precisely as 1/1000th of a second. Weylus then encodes frames as fast as possible and sets the timestamp accordingly: https://github.com/H-M-H/Weylus/blob/33cf5f138b5a7eefa8cf07c71eb0403a4fce849f/lib/encode_video.c#L476
Thank you for your support. I'm working on this and try to get encoding working with different parameters.
Is there a way to try the values for time_base, framerate and gop without recompiling the software? And if not, is there a way to only recompile parts of waylus without having to wait for the complete package?
It's possible but a little complicated. Instead I'd suggest abusing environment variables for testing. Once you are done you can remove this "hack". Just add something like
int gop_size = atoi(getenv("GOP_SIZE"));
to read a value from the environment. Then make sure to start Weylus like this GOP_SIZE=some_valid_integer_otherwise_this_results_in_undefined_behavior ./weylus.
I'm sorry for being delayed but I'm not a developer and even more not familiar with c code. I've tried to read the environment variables during runtime but it's not working. weylus is complaining that GOP_SIZE is not set correctly.
Can you please elaborate the necessary code changes a bit more. Sorry for my missing understanding and thank you in advance.
Don't worry. This is how you could do it, if you want to play with gop size and framerate:
diff --git a/lib/encode_video.c b/lib/encode_video.c
index e2f0a4c..8316b34 100644
--- a/lib/encode_video.c
+++ b/lib/encode_video.c
@@ -115,13 +115,19 @@ void init_ffmpeg_logger() { av_log_set_callback(log_callback); }
void set_codec_params(VideoContext* ctx)
{
+
+ int gop_size = atoi(getenv("GOP_SIZE"));
+ int frame_rate_num = atoi(getenv("FRAME_RATE_NUM"));
+ int frame_rate_den = atoi(getenv("FRAME_RATE_DEN"));
+
/* resolution must be a multiple of two */
ctx->c->width = ctx->width_out;
ctx->c->height = ctx->height_out;
ctx->c->time_base = (AVRational){1, 1000};
- ctx->c->framerate = (AVRational){0, 1};
+ ctx->c->framerate.num = frame_rate_num;
+ ctx->c->framerate.den = frame_rate_den;
- ctx->c->gop_size = 12;
+ ctx->c->gop_size = gop_size;
// no B-frames to reduce latency
ctx->c->max_b_frames = 0;
if (ctx->oc->oformat->flags & AVFMT_GLOBALHEADER)
Make sure to start Weylus with GOP_SIZE=12 FRAME_RATE_NUM=0 FRAME_RATE_DEN=1 ./weylus. Note that the framerate is given as fraction with 0/1 meaning unspecified / dynamic.
Thank you for the patch. I can build weylus with the patch and I've tried different settings for the newly created environment variables. Actually I have the impression that this is not the right direction because the error messages regarding GOP_SIZE is unchanged. I can see a changing number regarding fps but in chromium I get the error message that the video stream can't be decoded. To get closer to the root of the problem I need more logging information. I've tried setting the environment variable FFREPORT but I can't get ffmpeg to write a in a log file. Please explain a bit how to use the logging infrastructure of ffmpeg when ffmpeg is started bey weylus. I've found a report regarding a bug or missing feature in ffmpeg which can be the same reason as of for the problem with the video stream from weylus. To put a long storry short, one needs an way to set a parameter called REPEAT_SEQ_HEADER to true. I build ffmpeg with an patch so that ffmpeg can set the parameter. But now I need the same option in weylus to tell ffmpeg to set the parameter. I tried to add it as a variable in the code but with no luck. Pleas provide a patch for weylus to set the parameter in ffmpeg. Sorry for the hassle but I rely would love to use weylus with the raspberry pi :-)
You can get all ffmpeg log messages by increasing the log level of Weylus like this: WEYLUS_LOG_LEVEL="TRACE" weylus
Thanks for doing some more research! It looks like ffmpeg contains a bug which prevents proper encoding. There is a patch linked in the report. I included the patch in this branch, lets hope it works: https://github.com/H-M-H/Weylus/tree/rpi_v4l2
Thank you again for your effort and your time. I'm working with the new rpi branch. Today my sd-card with the 32-bit os died so I switched to the sd-card with an 64-bit os.
Actually on the 64-bit raspbian I get an linker error in the last step of building weylus. At the moment a build with the "--release" option is running and I'll report back if this is successful.
The error persists:
error: linking with `cc` failed: exit code: 1
|
= note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-Wl,--eh-frame-hdr" "-L" "/usr/lib/rustlib/aarch64-unknown-linux-gnu/lib" "/usr/local/src/Weylus-rpi/target/release/deps/weylus-208f1be6fc8e3d76.weylus.t6cl4kv5-c
gu.1.rcgu.o" "-o" "/usr/local/src/Weylus-rpi/ta`
...
-lutil" "-ldl" "-lutil" "-latomic"
= note: /usr/bin/ld: cannot find -lavdevice
/usr/bin/ld: cannot find -lavfilter
/usr/bin/ld: cannot find -lpostproc
/usr/bin/ld: cannot find -lx264
collect2: error: ld returned 1 exit status
error: aborting due to previous error; 1 warning emitted
error: could not compile `weylus`.
I believe this is related to the rust or cargo version. Unfortunately I'm not sure how to update to a newer version within rasbian.
Update I'll try to update with rustup. Let's see... Update II I have now cargo version "1.58.0 (f01b232bc 2022-01-19)" but no luck.
I tried with
RUSTFLAGS="-C linker=aarch64-linux-gnu-gcc" cargo build --release --target aarch64-unknown-linux-gnu --features ffmpeg-system
But I get almost the same error:
error: linking with `aarch64-linux-gnu-gcc` failed: exit status: 1
|
= note: "aarch64-linux-gnu-gcc" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/deps/weylus-242ab4e25ba5b36c.weylus.cc441366-cgu.6.rcgu.o" "-Wl,--as-needed" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/deps" "-L" "/usr/local/src/Weylus-rpi/target/release/deps" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/build/weylus-676eeed973b14fb1/out" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/build/weylus-676eeed973b14fb1/out" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/build/weylus-676eeed973b14fb1/out" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/build/weylus-676eeed973b14fb1/out" "-L" "/usr/lib/aarch64-linux-gnu" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/build/fltk-sys-eda75251aacf962d/out/build" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/build/fltk-sys-eda75251aacf962d/out/build/Release" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/build/fltk-sys-eda75251aacf962d/out/lib" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/build/fltk-sys-eda75251aacf962d/out/lib64" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/build/fltk-sys-eda75251aacf962d/out/lib/Release" "-L" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/build/fltk-sys-eda75251aacf962d/out/lib64/Release" "-L" "/root/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,--whole-archive" "-lerror" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-llog" "-Wl,--no-whole-archive" "-Wl,--whole-archive" "-lvideo" "-Wl,--no-whole-archive" "-Wl,-Bdynamic" "-lavcodec" "-lavdevice" "-lavfilter" "-lavformat" "-lavutil" "-lpostproc" "-lswresample" "-lswscale" "-lx264" "-Wl,-Bstatic" "-Wl,--whole-archive" "-llinux" "-Wl,--no-whole-archive" "-Wl,-Bdynamic" "-lX11" "-lXext" "-lXrandr" "-lXfixes" "-lXcomposite" "-lXi" "-lva" "-lva-drm" "-lva-x11" "-ldrm" "-Wl,-Bstatic" "/tmp/rustcHWVtNw/libfltk_sys-6d40b0c03894e2ae.rlib" "-Wl,--start-group" "-Wl,--end-group" "/root/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib/libcompiler_builtins-88a443e294197fb0.rlib" "-Wl,-Bdynamic" "-lgstvideo-1.0" "-lgstbase-1.0" "-lgstreamer-1.0" "-lgobject-2.0" "-lglib-2.0" "-lX11" "-lXtst" "-lpthread" "-lX11" "-lXext" "-lXinerama" "-lXcursor" "-lXrender" "-lXfixes" "-lXft" "-lfontconfig" "-lgstapp-1.0" "-lgstbase-1.0" "-lgstreamer-1.0" "-lgobject-2.0" "-lglib-2.0" "-lgstbase-1.0" "-lgstreamer-1.0" "-lgobject-2.0" "-lglib-2.0" "-lgstreamer-1.0" "-lgobject-2.0" "-lglib-2.0" "-lgobject-2.0" "-lglib-2.0" "-ldbus-1" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/root/.rustup/toolchains/stable-aarch64-unknown-linux-gnu/lib/rustlib/aarch64-unknown-linux-gnu/lib" "-o" "/usr/local/src/Weylus-rpi/target/aarch64-unknown-linux-gnu/release/deps/weylus-242ab4e25ba5b36c" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs"
= note: /usr/bin/ld: cannot find -lavdevice
/usr/bin/ld: cannot find -lavfilter
/usr/bin/ld: cannot find -lpostproc
/usr/bin/ld: cannot find -lx264
collect2: error: ld returned 1 exit status
warning: `weylus` (bin "weylus") generated 1 warning
error: could not compile `weylus` due to previous error; 1 warning emitted
Update
same with ld
error: linking with `aarch64-linux-gnu-ld` failed: exit status: 1 ...
Looks like building ffmpeg failed. Please check the contents of deps/dist/lib/. Among other things the following libs should be present
libavcodec.a
libavdevice.a
libavfilter.a
libavformat.a
libavutil.a
libpostproc.a
libswresample.a
libswscale.a
libva.a
If they are not, navigate into deps and execute clean.sh and then build again.
As I understand the build process I can pas --features ffmpeg-system to the build command to use the system libraries? Is this the wrong way? Is it mandatory in the rpi_4vl branch to build ffmpeg in the weylus build process?
Update In the deps directory is no dist directory. I've started a new build process without the "--features ffmpeg-system" option.
Oh, now I see the problem! Yes, it is mandatory as ffmpeg needs to be patched. The system version won't work.
Sorry but I thing the rpi branch is not ready. I see two issues
- In the encode_video.c the call for the h264_v4l2m2m is missing -> I added/ changed the code insted of libx264 I put h264_...
- The build process for ffmpeg is missing the h264_v4l2m2m driver So I tried the rpi branch but with no success.
My next attempt was to recompile ffmpeg out of the debian source package and before compiling I patched v4l2_m2m_enc.c as suggested in the report (it's the same what the patch in the weylus internal build of ffmpeg does). I got the ffmpeg deb packages and installed ffmpeg and libavcodec.
To start with the good part: The patch works and I get "Encoder: repeat parameter sets = 1" in the weylus trace. The bad part I still have no video and a new error:
2022-02-23T10:10:27.189023Z INFO weylus::log: Using device /dev/video11
2022-02-23T10:10:27.189434Z INFO weylus::log: driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
2022-02-23T10:10:27.189579Z INFO weylus::log: requesting formats: output=YU12 capture=H264
2022-02-23T10:10:27.215520Z TRACE weylus::log: output: YU12 16 buffers initialized: 1124x0632, sizeimage 01092096, bytesperline 00001152
2022-02-23T10:10:27.221552Z TRACE weylus::log: capture: H264 04 buffers initialized: 1124x0632, sizeimage 00552960, bytesperline 00000000
2022-02-23T10:10:27.221896Z TRACE weylus::log: Failed to set number of B-frames: Invalid argument
2022-02-23T10:10:27.222071Z TRACE weylus::log: Failed to get number of B-frames
2022-02-23T10:10:27.222268Z WARN weylus::log: Failed to set timeperframe
2022-02-23T10:10:27.222592Z TRACE weylus::log: Encoder: header mode = 0
2022-02-23T10:10:27.222874Z TRACE weylus::log: Encoder: bit rate = 200000
2022-02-23T10:10:27.223056Z TRACE weylus::log: Failed to set frame level rate control: Invalid argument
2022-02-23T10:10:27.223221Z WARN weylus::log: Failed to set gop size: Invalid argument
2022-02-23T10:10:27.223488Z TRACE weylus::log: Encoder: repeat parameter sets = 1
2022-02-23T10:10:27.223644Z TRACE weylus::log: Encoder Context: id (27), profile (-99), frame rate(0/1), number b-frames (0), gop size (12), bit rate (200000), qmin (-1), qmax (-1)
2022-02-23T10:10:27.223929Z TRACE weylus::log: Encoder: minimum video quantizer scale = 0
2022-02-23T10:10:27.224410Z TRACE weylus::log: Encoder: maximum video quantizer scale = 51
2022-02-23T10:10:27.225093Z DEBUG weylus::log: Empty MOOV enabled; disabling automatic bitstream filtering
2022-02-23T10:10:27.225614Z DEBUG weylus::log: Not writing 'clli' atom. No content light level info.
2022-02-23T10:10:27.225836Z DEBUG weylus::log: Not writing 'mdcv' atom. Missing mastering metadata.
2022-02-23T10:10:27.229606Z INFO weylus::log: Video: 1124x632@h264_v4l2m2m pix_fmt: yuv420p
2022-02-23T10:10:32.344507Z ERROR weylus::log: Application provided invalid, non monotonically increasing dts to muxer in stream 0: 448560 >= 0
2022-02-23T10:10:37.245047Z ERROR weylus::log: Application provided invalid, non monotonically increasing dts to muxer in stream 0: 889650 >= 0
2022-02-23T10:10:42.185585Z ERROR weylus::log: Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1335330 >= 0
2022-02-23T10:10:47.063446Z ERROR weylus::log: Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1774890 >= 0
2022-02-23T10:10:51.928700Z ERROR weylus::log: Application provided invalid, non monotonically increasing dts to muxer in stream 0: 2212740 >= 0
^C
Any suggestions on that invalid, non monotonically ... ?
Here is the output of v4l2-ctl -L -d /dev/video11 |more
`Codec Controls
video_bitrate_mode 0x009909ce (menu) : min=0 max=1 default=0 value=0 flags=update
0: Variable Bitrate
1: Constant Bitrate
video_bitrate 0x009909cf (int) : min=25000 max=25000000 step=25000 default=10000000 value=10000000
sequence_header_mode 0x009909d8 (menu) : min=0 max=1 default=1 value=1
0: Separate Buffer
1: Joined With 1st Frame
repeat_sequence_header 0x009909e2 (bool) : default=0 value=0
force_key_frame 0x009909e5 (button) : flags=write-only, execute-on-write
h264_minimum_qp_value 0x00990a61 (int) : min=0 max=51 step=1 default=20 value=20
h264_maximum_qp_value 0x00990a62 (int) : min=0 max=51 step=1 default=51 value=51
h264_i_frame_period 0x00990a66 (int) : min=0 max=2147483647 step=1 default=60 value=60
h264_level 0x00990a67 (menu) : min=0 max=13 default=11 value=11
0: 1
1: 1b
2: 1.1
3: 1.2
4: 1.3
5: 2
6: 2.1
7: 2.2
8: 3
9: 3.1
10: 3.2
11: 4
12: 4.1
13: 4.2
h264_profile 0x00990a6b (menu) : min=0 max=4 default=4 value=4
0: Baseline
1: Constrained Baseline
2: Main
2: Main
4: High`
I'm still working on a way to use Weylus with the raspberry pi but without success.
What I tried: switched to raspbian 32 bit build ffmpeg out of the sources with the patch build weylus
These are the parameters I set for weylus
/* resolution must be a multiple of two */
ctx->c->width = ctx->width_out;
ctx->c->height = ctx->height_out;
ctx->c->time_base = (AVRational){1, 100};
//ctx->c->time_base = -1;
ctx->c->framerate = (AVRational){1, 25};
//ctx->c->framerate = 120;
//av_opt_set(ctx->c->priv_data, "qp", "23", 0);
//av_opt_set(ctx->c->priv_data, "sequence_header_mode", "1", 0);
//ctx->c->gop_size = 12;
// no B-frames to reduce latency
ctx->c->max_b_frames = 0;
This is the log file:
2022-03-21T22:19:17.912080Z TRACE weylus::log: driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
2022-03-21T22:19:17.912699Z INFO weylus::log: Using device /dev/video11
2022-03-21T22:19:17.913272Z INFO weylus::log: driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
2022-03-21T22:19:17.913783Z INFO weylus::log: requesting formats: output=YU12 capture=H264
2022-03-21T22:19:17.932465Z TRACE weylus::log: output: YU12 16 buffers initialized: 1024x0768, sizeimage 01179648, bytesperline 00001024
2022-03-21T22:19:17.945280Z TRACE weylus::log: capture: H264 04 buffers initialized: 1024x0768, sizeimage 00589824, bytesperline 00000000
2022-03-21T22:19:17.949669Z TRACE weylus::log: Failed to set number of B-frames: Invalid argument
2022-03-21T22:19:17.950267Z TRACE weylus::log: Failed to get number of B-frames
2022-03-21T22:19:17.950966Z TRACE weylus::log: Encoder: header mode = 0
2022-03-21T22:19:17.951585Z TRACE weylus::log: Encoder: bit rate = 200000
2022-03-21T22:19:17.952106Z TRACE weylus::log: Failed to set frame level rate control: Invalid argument
2022-03-21T22:19:17.952621Z WARN weylus::log: Failed to set gop size: Invalid argument
2022-03-21T22:19:17.953116Z TRACE weylus::log: Encoder Context: id (27), profile (-99), frame rate(1/25), number b-frames (0), gop size (12), bit rate (200000), qmin (-1), qmax (-1)
2022-03-21T22:19:17.953715Z TRACE weylus::log: Encoder: minimum video quantizer scale = 0
2022-03-21T22:19:17.954243Z TRACE weylus::log: Encoder: maximum video quantizer scale = 51
2022-03-21T22:19:17.954969Z DEBUG weylus::log: Empty MOOV enabled; disabling automatic bitstream filtering
2022-03-21T22:19:17.955707Z DEBUG weylus::log: Not writing 'clli' atom. No content light level info.
2022-03-21T22:19:17.956231Z DEBUG weylus::log: Not writing 'mdcv' atom. Missing mastering metadata.
2022-03-21T22:19:17.959837Z INFO weylus::log: Video: 1024x768@h264_v4l2m2m pix_fmt: yuv420p
2022-03-21T22:19:18.007633Z TRACE weylus::log: --- output pre VIDIOC_QBUF: index 0, ts=1.170000 count=0
2022-03-21T22:19:18.008137Z TRACE weylus::log: --- output VIDIOC_QBUF: index 0, ts=1.170000 count=1
2022-03-21T22:19:18.014342Z TRACE weylus::log: output set status 1074026002 (ON) OK
2022-03-21T22:19:18.014733Z TRACE weylus::log: --- capture VIDIOC_QBUF: index 0, ts=0.000000 count=1
2022-03-21T22:19:18.015091Z TRACE weylus::log: --- capture VIDIOC_QBUF: index 1, ts=0.000000 count=2
2022-03-21T22:19:18.015449Z TRACE weylus::log: --- capture VIDIOC_QBUF: index 2, ts=0.000000 count=3
2022-03-21T22:19:18.015770Z TRACE weylus::log: --- capture VIDIOC_QBUF: index 3, ts=0.000000 count=4
2022-03-21T22:19:18.016510Z TRACE weylus::log: capture set status 1074026002 (ON) OK
2022-03-21T22:19:18.017128Z TRACE weylus::log: --- capture VIDIOC_DQBUF OK: index=0, ts=0.000000, count=3, dq=1 field=1
2022-03-21T22:19:18.017462Z INFO weylus::log: ff_v4l2_buffer_buf_to_avpkt
2022-03-21T22:19:18.017818Z WARN weylus::log: Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
2022-03-21T22:19:18.018192Z WARN weylus::log: Encoder did not produce proper pts, making some up.
2022-03-21T22:19:18.018581Z TRACE weylus::log: capture: Buffer requeue
2022-03-21T22:19:18.018988Z TRACE weylus::log: --- capture VIDIOC_QBUF: index 0, ts=0.000000 count=4
2022-03-21T22:19:18.076756Z TRACE weylus::log: --- output VIDIOC_DQBUF OK: index=0, ts=1.170000, count=0, dq=1 field=1
2022-03-21T22:19:18.077796Z TRACE weylus::log: --- output pre VIDIOC_QBUF: index 0, ts=1.870000 count=0
2022-03-21T22:19:18.078309Z TRACE weylus::log: --- output VIDIOC_QBUF: index 0, ts=1.870000 count=1
2022-03-21T22:19:18.078712Z TRACE weylus::log: --- capture VIDIOC_DQBUF OK: index=1, ts=1.170000, count=3, dq=2 field=1
2022-03-21T22:19:18.079038Z INFO weylus::log: ff_v4l2_buffer_buf_to_avpkt
2022-03-21T22:19:18.079666Z TRACE weylus::log: capture: Buffer requeue
2022-03-21T22:19:18.080074Z TRACE weylus::log: --- capture VIDIOC_QBUF: index 1, ts=1.170000 count=4
2022-03-21T22:19:18.138675Z TRACE weylus::log: --- output VIDIOC_DQBUF OK: index=0, ts=1.870000, count=0, dq=2 field=1
2022-03-21T22:19:18.139676Z TRACE weylus::log: --- output pre VIDIOC_QBUF: index 0, ts=2.490000 count=0
2022-03-21T22:19:18.140158Z TRACE weylus::log: --- output VIDIOC_QBUF: index 0, ts=2.490000 count=1
2022-03-21T22:19:18.140504Z TRACE weylus::log: --- capture VIDIOC_DQBUF OK: index=2, ts=1.870000, count=3, dq=3 field=1
2022-03-21T22:19:18.140829Z INFO weylus::log: ff_v4l2_buffer_buf_to_avpkt
Any help is appreciated.
Thanks for further investigating here @saturn-hh ! I'd recommend trying to get encoding with the ffmpeg CLI working first. Either try the patched version of ffmpeg you compiled yourself or use the one that is produced as build artifact at deps/dist/bin/ffmpeg (I updated https://github.com/H-M-H/Weylus/tree/rpi_v4l2 to build the ffmpeg binary with v4l2 support and added some code to make Weylus try to use said codec (should have done that in the first place)).
This invocation of ffmpeg is pretty close to what Weylus does, check if you can get this working: ./ffmpeg -f lavfi -i testsrc2=size=1920x1080 -vcodec h264_v4l2m2m -t 10 -bf 0 -movflags frag_keyframe+empty_moov+default_base_moof test.mp4 and if the produced video is playable by your browser.
I hope this is at least of some help!
I've tried the command you mentioned and at the end of the encoding process I got a file and no error message, BUT the file isn't playable. When trying to play with ffplay I get: "No start code is found."
I found this post witch is maybe related to the issue.
Investigating further with ffprobe give me this report:
probe started on 2022-03-24 at 15:57:09
Report written to "ffprobe-20220324-155709.log"
Log level: 48
Command line:
ffprobe -report -loglevel debug 20220324-test.mp4
ffprobe version a4e1dd6 Copyright (c) 2007-2021 the FFmpeg developers
built with gcc 10 (Raspbian 10.2.1-6+rpi1)
configuration: --extra-cflags=-I/opt/vc/include/IL --extra-ldflags=-L/usr/local/lib --extra-libs='-lpthread -lm -latomic' --arch=armel --enable-gmp --enable-gpl --enable-libaom --enable-libass --enable-libdav1d --enable-libdrm --e
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
[NULL @ 0x33ae280] Opening '20220324-test.mp4' for reading
[file @ 0x33aeaf0] Setting default whitelist 'file,crypto,data'
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] Format mov,mp4,m4a,3gp,3g2,mj2 probed with size=2048 and score=100
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] ISO: File Type Major Brand: iso5
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] Unknown dref type 0x206c7275 size 12
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] found tfdt time 0, using it for dts
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] found tfdt time 1, using it for dts
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] found tfdt time 30720, using it for dts
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] found tfdt time 61440, using it for dts
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] found tfdt time 92160, using it for dts
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] found tfdt time 122880, using it for dts
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] Before avformat_find_stream_info() pos: 1453852 bytes read:184930 seeks:4 nb_streams:1
[extract_extradata @ 0x33d8600] No start code is found.
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x33ae280] After avformat_find_stream_info() pos: 910 bytes read:217698 seeks:5 frames:0
20220324-test.mp4: Invalid data found when processing input
[AVIOContext @ 0x33aec90] Statistics: 217698 bytes read, 5 seeks
This is the report of the encoding process:
Report written to "20220324-ffreport-movflags.log"
Log level: 48
Command line:
ffmpeg -f lavfi -i "testsrc2=size=1920x1080" -vcodec h264_v4l2m2m -t 10 -bf 0 -movflags frag_keyframe+empty_moov+default_base_moof 20220324-test.mp4
ffmpeg version a4e1dd6 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 10 (Raspbian 10.2.1-6+rpi1)
configuration: --extra-cflags=-I/opt/vc/include/IL --extra-ldflags=-L/usr/local/lib --extra-libs='-lpthread -lm -latomic' --arch=armel --enable-gmp --enable-gpl --enable-libaom --enable-libass --enable-libdav1d --enable-libdrm --enable-libfdk-aac --enable-libfreetype --enable-libkvazaar --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enabl
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
Splitting the commandline.
Reading option '-f' ... matched as option 'f' (force format) with argument 'lavfi'.
Reading option '-i' ... matched as input url with argument 'testsrc2=size=1920x1080'.
Reading option '-vcodec' ... matched as option 'vcodec' (force video codec ('copy' to copy stream)) with argument 'h264_v4l2m2m'.
Reading option '-t' ... matched as option 't' (record or transcode "duration" seconds of audio/video) with argument '10'.
Reading option '-bf' ... matched as AVOption 'bf' with argument '0'.
Reading option '-movflags' ... matched as AVOption 'movflags' with argument 'frag_keyframe+empty_moov+default_base_moof'.
Reading option '20220324-test.mp4' ... matched as output url.
Finished splitting the commandline.
Parsing a group of options: global .
Successfully parsed a group of options.
Parsing a group of options: input url testsrc2=size=1920x1080.
Applying option f (force format) with argument lavfi.
Successfully parsed a group of options.
Opening an input file: testsrc2=size=1920x1080.
detected 4 logical cores
[Parsed_testsrc2_0 @ 0x2454d20] Setting 'size' to value '1920x1080'
[Parsed_testsrc2_0 @ 0x2454d20] size:1920x1080 rate:25/1 duration:-1.000000 sar:1/1
[AVFilterGraph @ 0x2453050] query_formats: 2 queried, 1 merged, 0 already done, 0 delayed
[lavfi @ 0x2452480] All info found
Input #0, lavfi, from 'testsrc2=size=1920x1080':
Duration: N/A, start: 0.000000, bitrate: N/A
Stream #0:0, 1, 1/25: Video: rawvideo (I420 / 0x30323449), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 25 tbr, 25 tbn, 25 tbc
Successfully opened the file.
Parsing a group of options: output url 20220324-test.mp4.
Applying option vcodec (force video codec ('copy' to copy stream)) with argument h264_v4l2m2m.
Applying option t (record or transcode "duration" seconds of audio/video) with argument 10.
Successfully parsed a group of options.
Opening an output file: 20220324-test.mp4.
[file @ 0x245a8d0] Setting default whitelist 'file,crypto,data'
Successfully opened the file.
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (h264_v4l2m2m))
Press [q] to stop, [?] for help
cur_dts is invalid st:0 (0) [init:0 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[rawvideo @ 0x2457f30] PACKET SIZE: 3110400, STRIDE: 2880
[graph 0 input from stream 0:0 @ 0x245c5a0] Setting 'video_size' to value '1920x1080'
[graph 0 input from stream 0:0 @ 0x245c5a0] Setting 'pix_fmt' to value '0'
[graph 0 input from stream 0:0 @ 0x245c5a0] Setting 'time_base' to value '1/25'
[graph 0 input from stream 0:0 @ 0x245c5a0] Setting 'pixel_aspect' to value '1/1'
[graph 0 input from stream 0:0 @ 0x245c5a0] Setting 'frame_rate' to value '25/1'
[graph 0 input from stream 0:0 @ 0x245c5a0] w:1920 h:1080 pixfmt:yuv420p tb:1/25 fr:25/1 sar:1/1
[AVFilterGraph @ 0x245bbf0] query_formats: 4 queried, 3 merged, 0 already done, 0 delayed
[h264_v4l2m2m @ 0x24598b0] probing device /dev/video18
[h264_v4l2m2m @ 0x24598b0] driver 'bcm2835-codec' on card 'bcm2835-codec-image_fx' in mplane mode
[h264_v4l2m2m @ 0x24598b0] v4l2 capture format not supported
[h264_v4l2m2m @ 0x24598b0] probing device /dev/video12
[h264_v4l2m2m @ 0x24598b0] driver 'bcm2835-codec' on card 'bcm2835-codec-isp' in mplane mode
[h264_v4l2m2m @ 0x24598b0] v4l2 capture format not supported
[h264_v4l2m2m @ 0x24598b0] probing device /dev/video11
[h264_v4l2m2m @ 0x24598b0] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x24598b0] Using device /dev/video11
[h264_v4l2m2m @ 0x24598b0] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x24598b0] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x24598b0] output: YU12 16 buffers initialized: 1920x1080, sizeimage 03110400, bytesperline 00001920
[h264_v4l2m2m @ 0x24598b0] capture: H264 04 buffers initialized: 1920x1080, sizeimage 01568768, bytesperline 00000000
[h264_v4l2m2m @ 0x24598b0] Failed to set number of B-frames: Invalid argument
[h264_v4l2m2m @ 0x24598b0] Failed to get number of B-frames
[h264_v4l2m2m @ 0x24598b0] Encoder: header mode = 0
[h264_v4l2m2m @ 0x24598b0] Encoder: bit rate = 200000
[h264_v4l2m2m @ 0x24598b0] Failed to set frame level rate control: Invalid argument
[h264_v4l2m2m @ 0x24598b0] Encoder: repeat parameter sets = 1
[h264_v4l2m2m @ 0x24598b0] Encoder Context: id (27), profile (-99), frame rate(25/1), number b-frames (0), gop size (12), bit rate (200000), qmin (-1), qmax (-1)
[h264_v4l2m2m @ 0x24598b0] Encoder: minimum video quantizer scale = 0
[h264_v4l2m2m @ 0x24598b0] Encoder: maximum video quantizer scale = 51
[mp4 @ 0x24582d0] Empty MOOV enabled; disabling automatic bitstream filtering
Output #0, mp4, to '20220324-test.mp4':
Metadata:
encoder : Lavf58.76.100
Stream #0:0, 0, 1/12800: Video: h264 (avc1 / 0x31637661), yuv420p(progressive), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 25 fps, 12800 tbn
Metadata:
encoder : Lavc58.134.100 h264_v4l2m2m
Clipping frame in rate conversion by 0.000008
frame= 1 fps=0.0 q=0.0 size= 0kB time=00:00:00.00 bitrate=N/A speed= 0x Mcur_dts is invalid st:0 (0) [init:1 i_done:0 finish:0] (this is harmless if it occurs once at the start per stream)
[rawvideo @ 0x2457f30] PACKET SIZE: 3110400, STRIDE: 2880
[rawvideo @ 0x2457f30] PACKET SIZE: 3110400, STRIDE: 2880
[mp4 @ 0x24582d0] Non-monotonous DTS in output stream 0:0; previous: 0, current: 0; changing to 1. This may result in incorrect timestamps in the output file.
...#
[out_0_0 @ 0x245cbd0] EOF on sink link out_0_0:default.
No more output streams to write to, finishing.
frame= 250 fps= 45 q=-0.0 Lsize= 1420kB time=00:00:09.96 bitrate=1167.7kbits/s speed=1.81x
video:1416kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.246505%
Input file #0 (testsrc2=size=1920x1080):
Input stream #0:0 (video): 251 packets read (780710400 bytes); 251 frames decoded;
Total: 251 packets (780710400 bytes) demuxed
Output file #0 (20220324-test.mp4):
Output stream #0:0 (video): 250 frames encoded; 255 packets muxed (1450277 bytes);
Total: 255 packets (1450277 bytes) muxed
251 frames successfully decoded, 0 decoding errors
[AVIOContext @ 0x245a980] Statistics: 0 seeks, 7 writeouts
BUT the file isn't playable.
Does this mean no video at all or a corrupted file like in the linked issue?
Please check if this fails too: ./ffmpeg -f lavfi -i testsrc2=size=1920x1080 -vcodec h264_v4l2m2m -t 10 test.mp4
If it does and you can't get ffmpeg itself to properly encode a test video, I'd suggest asking the people at the #ffmpeg irc channel on libera.chat.
We are going in the right direction! First time that I got a picture in the browser.
The key is to change the movflags. Without the flag empty_moov its working.
Patching ffmpeg or libavcodec is NOT neccessary. System is an Raspberry pi OS 64 bit bullyeys with packages from the repos.
Picture is choppy but I'll try some tuning.
Cheers Allexander
Nice! In that case adjusting https://github.com/H-M-H/Weylus/blob/31780cd1965b1819e3b2bf3acb63d9d8cf583926/lib/encode_video.c#L441 might be enough to get Weylus to stream a decodable video.
I can't make it happen. Probably there is a regression in the actual version of h264_v4l2m2m in 32 and 64 bit version. I haven't found a way to produce a valid fragmented mp4 file. With the movflag "empty_moov" the mp4 is not playable and without this movflag the steam given by weylus to the browser is not playable. I've tried different combinations of movflags here.
I've opened a report here and here but unfortunately none responded.
In the depreciated version of Rasbian Buster I can produce a correct mp4 file but I can't compile weylus. No matter what I try, in the final step I get this linker error:
note: /usr/local/bin/ld: /usr/local/src/Weylus/target/debug/deps/libfltk_sys-9948ac0e1fb1f87b.rlib: error adding symbols: file format not recognized
I've updated gcc and ld manually but the error persists.
So now I'm out of options.
Any ideas about this?
Hi. Instead of RPi, I am working with a Jetson Nano 2G. With a modified version of ffmpeg, ffmpeg -f lavfi -i testsrc2=size=1920x1080 -vcodec h264_nvmpi -t 10 -bf 0 -movflags frag_keyframe+empty_moov+default_base_moof test_encode.mp4 runs perfectly. However before anything specific to the codec comes up, after replacing libx264 with h264_nvmpi and compiling Weylus, I am seeing Codec 'h264_nvmpi' not found being printed. I can confirm that the codec name is correct since it's defined here as
#define NVMPI_ENC(NAME, LONGNAME, CODEC) \
NVMPI_ENC_CLASS(NAME) \
FFCodec ff_ ## NAME ## _nvmpi_encoder = { \
.p.name = #NAME "_nvmpi" , \
CODEC_LONG_NAME("nvmpi " LONGNAME " encoder wrapper"), \
.p.type = AVMEDIA_TYPE_VIDEO, \
.p.id = CODEC , \
.priv_data_size = sizeof(nvmpiEncodeContext), \
.p.priv_class = &nvmpi_ ## NAME ##_enc_class, \
.init = nvmpi_encode_init, \
FF_CODEC_ENCODE_CB(nvmpi_encode_frame), \
.close = nvmpi_encode_close, \
.p.pix_fmts = (const enum AVPixelFormat[]) { AV_PIX_FMT_YUV420P, AV_PIX_FMT_NONE },\
.p.capabilities = AV_CODEC_CAP_HARDWARE | AV_CODEC_CAP_DELAY, \
.defaults = defaults,\
.p.wrapper_name = "nvmpi", \
};
NVMPI_ENC(h264, "H.264", AV_CODEC_ID_H264);
NVMPI_ENC(hevc, "HEVC", AV_CODEC_ID_HEVC);
Edit: I find out I was building a static library but asking weylus to use the system shared library. I am rebuilding with the static library to see if there any problem persists.
Edit: I decided to build ffmpeg as shared library instead and this time it works, although the performance is not good. I am still tuning the parameters to see if I can get any improvement.
