rpicam-apps
rpicam-apps copied to clipboard
raspberrypi
[BUG] Crashes and errors when recording video+audio with libcamera-vid/libav on RPI ZERO (Bullseye)
Hello,
I've tried all kinds of combinations to record video+audio on a raspberry pi zero V1, with libcamera-vid.
I always get "Application provided invalid, non monotonically increasing dts to muxer in stream" or other errors (see below).
I've tried all kinds of microphones (MEMS, USB, Waveshare audio pHat...), which all work well with "arecord" but fail with libcamera+libav.
There are options in ffmpeg to workaround some problems like invalid DTS, but it doesn't seem that I can add these options from that CLI.
Since libcamera-vid directly records from Alsa device, I don't get why DTS would be invalid, however.
Performance-wise, recording doesn't seem to saturate CPU or RAM (it's more like 50% of CPU).
Below are the ouputs of the 2 commands, and attached are the camera bug reports of similar launches.
Am I missing something ? ^^'
(.venv) pi@raspberrypi:~/ $ libcamera-vid --timeout 60 --nopreview --output file.mp4 --codec libav --libav-audio
[8:09:56.126512784] [2316] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
[8:09:57.158350223] [2317] INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media0
[8:09:57.194793920] [2316] INFO Camera camera.cpp:1035 configuring streams: (0) 640x480-YUV420
[8:09:57.202986852] [2317] INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
[h264_v4l2m2m @ 0x11b6320] <<< v4l2_encode_init: fmt=181/0
[h264_v4l2m2m @ 0x11b6320] Using device /dev/video11
[h264_v4l2m2m @ 0x11b6320] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x11b6320] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x11b6320] Failed to set number of B-frames: Invalid argument
[h264_v4l2m2m @ 0x11b6320] Failed to set gop size: Invalid argument
Input #0, pulse, from 'default':
Duration: N/A, start: 1665587747.284031, bitrate: 1536 kb/s
Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Output #0, mp4, to 'file.mp4':
Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 1000k tbn
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s
Halting: reached timeout of 60 milliseconds.
[h264_v4l2m2m @ 0x11b6320] V4L2 capture poll unexpected timeout: events=0x2
Segmentation fault
(.venv) pi@raspberrypi:~/ $ libcamera-vid --timeout 0 --nopreview --output file.mpeg --codec libav --libav-format mpegts --libav-audio
[8:15:07.002135343] [2347] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
[8:15:07.936420591] [2348] INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media0
[8:15:07.964614357] [2347] INFO Camera camera.cpp:1035 configuring streams: (0) 640x480-YUV420
[8:15:07.967584333] [2348] INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
[h264_v4l2m2m @ 0x13ec410] <<< v4l2_encode_init: fmt=181/0
[h264_v4l2m2m @ 0x13ec410] Using device /dev/video11
[h264_v4l2m2m @ 0x13ec410] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x13ec410] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x13ec410] Failed to set number of B-frames: Invalid argument
[h264_v4l2m2m @ 0x13ec410] Failed to set gop size: Invalid argument
Input #0, pulse, from 'default':
Duration: N/A, start: 1665588061.137827, bitrate: 1536 kb/s
Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Output #0, mpegts, to 'file.mpeg':
Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 1000k tbn
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s
[pulse @ 0x13f6d10] pa_stream_get_latency() failed
[pulse @ 0x13f6d10] pa_stream_get_latency() failed
[...]
[pulse @ 0x13f6d10] pa_stream_get_latency() failed
[pulse @ 0x13f6d10] pa_stream_get_latency() failed
[aac @ 0x13fe280] Queue input is backward in time
[aac @ 0x13fe280] Queue input is backward in time
[aac @ 0x13fe280] Queue input is backward in time
[mpegts @ 0x13ec7c0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 55312 >= 41991
terminate called after throwing an instance of 'std::runtime_error'
what(): libav: error writing output: -22
Aborted
bug_report_mp4_libcamera_audio.txt bug_report_mpegts_libcamera_audio.txt
@pakal thank you for your report. I don't recall seeing any of these errors before during my runs - but I have used a Pi 4 device not a Pi Zero. Are you able to try exactly the same commands on a Pi 3B+ or 4 to see if you get the same problems?
I've just tested with same camera+soundcard on a Raspberry pi 3+ with a new OS image, the recording of MP4 and MPEG performs fine.
So it would be a problem with the pi zero v1 hardware/firmware? (but CPU doesn't seem exceeded though)
I alas can't get my hand on a pi zero v2 to compare, everything is sold out.
I've retested the PI ZERO with a fresh bullseye, with standard camera, and USB soundcard from waveshare.
The MPEGTS recording now does WORK with "libcamera-vid -t 600000 --nopreview -o output4_pizero.mpeg --codec libav --libav-format=mpegts --libav-audio" ; so maybe there would be bad interactions with the few overlays (for RTC clock, waveshare HAT, I2S microphone...) that I had installed on my previous bullseye setup ?
But the MP4 recording still fails, with both old and new errors (randomly), so the DTS problem is still lurking in the recording system it seems :
pi@raspberrypi:~ $ libcamera-vid -t 600000 --nopreview -o output4_pizero.mp4 --codec libav --libav-format=mp4 --libav-audio
[0:57:04.838975182] [1518] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
[0:57:05.344500841] [1519] INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media0
[0:57:05.348882656] [1518] INFO Camera camera.cpp:1035 configuring streams: (0) 640x480-YUV420
[0:57:05.352113519] [1519] INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
[h264_v4l2m2m @ 0x15e4410] <<< v4l2_encode_init: fmt=181/0
[h264_v4l2m2m @ 0x15e4410] Using device /dev/video11
[h264_v4l2m2m @ 0x15e4410] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x15e4410] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x15e4410] Failed to set number of B-frames: Invalid argument
[h264_v4l2m2m @ 0x15e4410] Failed to set gop size: Invalid argument
Input #0, pulse, from 'default':
Duration: N/A, start: 1665675804.075169, bitrate: 1536 kb/s
Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Output #0, mp4, to 'output4_pizero.mp4':
Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 1000k tbn
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s
libcamera-vid: ../include/libcamera/controls.h:152: T libcamera::ControlValue::get() const [with T = int; typename std::enable_if<((! libcamera::details::is_span<U>::value) && (! std::is_same<std::__cxx11::basic_string<char>, typename std::remove_cv< <template-parameter-1-1> >::type>::value)), std::nullptr_t>::type <anonymous> = nullptr]: Assertion `type_ == details::control_type<std::remove_cv_t<T>>::value' failed.
Aborted
pi@raspberrypi:~ $ libcamera-vid -t 600000 --nopreview -o output4_pizero.mp4 --codec libav --libav-format=mp4 --libav-audio
[0:57:18.408183345] [1529] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
[0:57:18.936120057] [1530] INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media0
[0:57:18.940900856] [1529] INFO Camera camera.cpp:1035 configuring streams: (0) 640x480-YUV420
[0:57:18.943839732] [1530] INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
[h264_v4l2m2m @ 0xacf410] <<< v4l2_encode_init: fmt=181/0
[h264_v4l2m2m @ 0xacf410] Using device /dev/video11
[h264_v4l2m2m @ 0xacf410] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0xacf410] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0xacf410] Failed to set number of B-frames: Invalid argument
[h264_v4l2m2m @ 0xacf410] Failed to set gop size: Invalid argument
Input #0, pulse, from 'default':
Duration: N/A, start: 1665675817.619173, bitrate: 1536 kb/s
Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Output #0, mp4, to 'output4_pizero.mp4':
Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 1000k tbn
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s
[h264_v4l2m2m @ 0xacf410] Pkt tracking failure: pts=74, track[74]=202
[mp4 @ 0xacf7f0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
[mp4 @ 0xacf7f0] Encoder did not produce proper pts, making some up.
[h264_v4l2m2m @ 0xacf410] Pkt tracking failure: pts=75, track[75]=203
[h264_v4l2m2m @ 0xacf410] Pkt tracking failure: pts=76, track[76]=204
[h264_v4l2m2m @ 0xacf410] Pkt tracking failure: pts=77, track[77]=205
[h264_v4l2m2m @ 0xacf410] Pkt tracking failure: pts=78, track[78]=206
[h264_v4l2m2m @ 0xacf410] Pkt tracking failure: pts=79, track[79]=207
[h264_v4l2m2m @ 0xacf410] Pkt tracking failure: pts=80, track[80]=208
[h264_v4l2m2m @ 0xacf410] Pkt tracking failure: pts=81, track[81]=209
[h264_v4l2m2m @ 0xacf410] Pkt tracking failure: pts=82, track[82]=210
[h264_v4l2m2m @ 0xacf410] Pkt tracking failure: pts=83, track[83]=211
[aac @ 0xae1530] Queue input is backward in time
[mp4 @ 0xacf7f0] Application provided invalid, non monotonically increasing dts to muxer in stream 1: 210387 >= 210031
terminate called after throwing an instance of 'std::runtime_error'
what(): libav: error writing output: -22
Thanks for the update!
It's entirely possible that the Zero platform might be underpowered for this task - particularly saving mp4 as it's a much more complicated container. It may be why running those extra overlays tilts things over the edge. I'll get myself a Zero and try it myself.
In the meantime, are you able to add the following to /boot/config.txt and see if that improves things:
over_voltage=4
force_turbo=1
I've managed to try this on a Pi Zero now - things seem to work for me.
Note that I am running Raspberry Pi OS lite (so no GUI), and I have to pull in the correct libraries for libav as they are not available on the lite OS by default:
sudo apt update
sudo apt install libcamera-apps
sudo apt install pulseaudio
After a reboot, I recorded with your command:
libcamera-vid --timeout 15000 --nopreview --output file.mp4 --codec libav --libav-audio
Are you able to try the same with an OS lite image (if you are not already using this)?
Thanks for the suggestions !
I've tried the overclocking, but still got the same errors randomly (core voltage is 1.4V in that case).
I've booted to console to avoid the X stack (I'm not certain it's the same as "Lite" though, some extra overlays and services might still interfere), and got the error too.
I will try with a pristine Lite image like you, when I have some more time, but the problem is that I'll need a full Desktop distrib anyway, with the few driver overlays I've installed, for the software to work (there is a GUI frontend). Will be interesting to see if it's a software or performance issue though.
I have now managed to reproduce this issue - or at least I think it might be the same issue. Some containers seem to be more sensitive in how they handle audio timestamps (avi in particular, which is how i reproduced this).
I have reworked the timestamping code and hopefully it should solve the issue.
Potential fix available at ~~https://github.com/raspberrypi/libcamera-apps/pull/389~~ https://github.com/raspberrypi/libcamera-apps/pull/392.
EDIT: Got the wrong PR!
I think you mean https://github.com/raspberrypi/libcamera-apps/pull/392 ^^
Yes media streams always have issues with DTS and the like, in my experience - I had to fiddle with lots of flags like "-fflags", "+igndts", "-movflags", "empty_moov+delay_moov", when I dealt with RTSP streams in FFMPEG for example.
I'll test that PR asap
Ok I've tested the MPEGTS and MP4 formats with the new version, DTS errors seem to have disappeared :)
There are still "pa_stream_get_latency() failed" errors though, which seem to corrupt a little bit the mpegts , and corrupt a lot the mp4 (there are dozens of such errors for mp4, and the 60mn file is only 7KB so it seems video and audio are mostly broken).
pi@raspberrypi:~/licamera_naushir/build $ ./libcamera-vid -t 60000 --nopreview -o output_pizero_audiofixed.mpegts --codec libav --libav-format=mpegts --libav-audio
[1:28:37.839564128] [922] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
[1:28:38.099871257] [923] INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media0
[1:28:38.105029200] [922] INFO Camera camera.cpp:1035 configuring streams: (0) 640x480-YUV420
[1:28:38.107762170] [923] INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam form
[h264_v4l2m2m @ 0x1675ff0] <<< v4l2_encode_init: fmt=181/0
[h264_v4l2m2m @ 0x1675ff0] Using device /dev/video11
[h264_v4l2m2m @ 0x1675ff0] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x1675ff0] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x1675ff0] Failed to set number of B-frames: Invalid argument
[h264_v4l2m2m @ 0x1675ff0] Failed to set gop size: Invalid argument
Input #0, pulse, from 'default':
Duration: N/A, start: 1666107938.800486, bitrate: 1536 kb/s
Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Output #0, mpegts, to 'output_pizero_audiofixed.mpegts':
Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 30 fps, 30 tbr, 1000k tbn
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s
Halting: reached timeout of 60000 milliseconds.
[pulse @ 0x16807c0] pa_stream_get_latency() failed
[aac @ 0x1687950] Qavg: 111.737
pi@raspberrypi:~/licamera_naushir/build $
---------------
pi@raspberrypi:~/licamera_naushir/build $ ./libcamera-vid -t 60000 --nopreview -o output_pizero_audiofixed_bis.mp4 --codec libav --libav-format=mp4 --libav-audio
[1:44:17.487007740] [992] INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
[1:44:17.839997812] [994] INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media3 and ISP device /dev/media0
[1:44:17.845205755] [992] INFO Camera camera.cpp:1035 configuring streams: (0) 640x480-YUV420
[1:44:17.847912724] [994] INFO RPI raspberrypi.cpp:761 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
[h264_v4l2m2m @ 0x18b0220] <<< v4l2_encode_init: fmt=181/0
[h264_v4l2m2m @ 0x18b0220] Using device /dev/video11
[h264_v4l2m2m @ 0x18b0220] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0x18b0220] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0x18b0220] Failed to set number of B-frames: Invalid argument
[h264_v4l2m2m @ 0x18b0220] Failed to set gop size: Invalid argument
Input #0, pulse, from 'default':
Duration: N/A, start: 1666108879.254740, bitrate: 1536 kb/s
Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Output #0, mp4, to 'output_pizero_audiofixed_bis.mp4':
Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 30 fps, 30 tbr, 1000k tbn
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s
[pulse @ 0x18bb810] pa_stream_get_latency() failed
[pulse @ 0x18bb810] pa_stream_get_latency() failed
<............>
[pulse @ 0x18bb810] pa_stream_get_latency() failed
[pulse @ 0x18bb810] pa_stream_get_latency() failed
Halting: reached timeout of 60000 milliseconds.
[aac @ 0x18c29a0] Qavg: 106.871
There are still "pa_stream_get_latency() failed" errors though, which seem to corrupt a little bit the mpegts , and corrupt a lot the mp4 (there are dozens of such errors for mp4, and the 60mn file is only 7KB so it seems video and audio are mostly broken).
Yes, I can see this as well on my Pi Zero board. Unfortunately, I think this use case may simply be too much for the Pi Zero CPU core. It looks pegged to 100% through the entire run, which is not great. As a consequence it starts dropping frames and encoded packets. Removing audio encoding from your command line seem to produce a correct output.
I'll see what I can do to possibly reduce the load for the audio encode thread, but that may not really gain us much either.
Whups, and almost no ways to get a pi zero 2 currently it seems ^^'
I don't know well the underlying hardware behaviour, nor the alsa/pulsaudio stakcs, is there a way to reduce the load by diminishing video framerate, image dimension/colordepth/quality, or audio samplerate/bitrate ?
Or are the input parameters of the camera/microphone rather fixed for most of these settings, and it's the codec which will have extra work to encode the initial recording to a lower resolution/rate ? I'll have to do benchmarks regarding battery consumption with these different scenarios, too.
We still have the option to record audio and video separately, and merge later, but the offset will not be easy to handle, so being able to get a video+audio stream on pi zero, even low-quality would help in some cases.
Using the avi container seems to work without any issues for me:
./libcamera-vid -t 60000 --nopreview -o output_pizero_audiofixed_bis.avi --codec libav --libav-audio
I do have the following in my config.txt file:
force_turbo=1
over_voltage=6
Not sure if that helps or not though...
EDIT: Just tried encoding to avi without the above 2 lines in my config.txt, and it still works without errors.
Thanks for the info, I'll try this too.
For the real use I'll need to PIPE the stream, so only mpegts would do the job I guess ?
I think avi supports piping as it does not seek back in the stream - but you need to try to confirm. However, mpegts will definitely work with pipes.
EDIT: Seems like avi does support piping to stdout.
Is there anything more we need to look at for this issue? If not, I'll close this down shortly.
I've just been doing a bunch of tests this afternoon, precisely :)
I've setup: force_turbo=1 over_voltage=6
I've tried AVI recording with "./libcamera-vid -t 120000 --nopreview -o myfile3.avi --codec libav --libav-audio --libav-format=avi"
- it works better than other attempts, no more "pa_stream_get_latency() failed"
- piping does work, although the file appears "truncated" to MediaInfo analysis in this case, so the position control is buggy under videoplayer (I guess it's the way AVI format works, it needs some finalization of file)
- weirdly, the file always has these MediaInfo parameters, even if I forced "--framerate=15" while recording (why the 600fps?) Frame rate : 600.000 FPS Original frame rate : 30.000 FPS
- the only real problem: audio becomes fully blank after 30 to 60 seconds (not silent, it completely stops, as seen under Audacity) ; with "arecord", though, no problem to recorder several minutes of audio from the same source. It seems that audio stops at HALF of the requested "--timeout" (ex. sound stops at 60s for a 2mn video).
I've tried again with MPEGTS:
- no more error either, so "over_voltage=6" instead of "over_voltage=4" does seem to help.
- no problem with seek'ing videos pipes with mpegts.
- framerate is set to 30 in mpeg file even why I force it to 15 with "--framerate" (but no weird "600fps" value)
- same bug with the audio stopping at half of the video file.
So it almost works, but the sound recording which stops halfway kinda ruins the shot ^^
PS: when recording at 2 fps, it clearly shows that the video is really low-quality, but it's the metadata which is somehow wrong, when read by vlc or mediaplayerclassic or least (they think it's 30fps).
The avi framerate error has already been fixed by b89cc36c3872a8ae711b6ff6fae0bb77b6dfe927. If you pull the latest code tree, that should be fixed.
I'll try to reproduce the audio stopping after a while during recording. I can't recall how long my longest recording was, but I did not notice this problem.
Trying to record a 2 min clip with the following command on a PI4 seems to produce a correct output, i.e. audio stream present till the end:
libcamera-vid --nopreview --codec libav -t 0 -o test.ts --libav-format mpegts --framerate 30 -v --libav-audio
I'll get my hands on a Pi Zero and try this same at some point. Can you share the command line you have been using? Can you also check if you have commit 05daa7948b1208f4d77b54c47bd43159c7d5dc15 in your tree? That does fix an issue with audio timestamping.
I've been using this command line, quite similar to yours, on pi zero (and it gave truncated audio track) :
"libcamera-vid -t 120000 --nopreview -o myfile3.avi --codec libav --libav-audio --libav-format=avi"
I've just setup a raspberry pi 3b+ with Bullseye, it'll help me compare the behaviours with the pi zero too ^^
@pakal I wonder if you could try out some experimental changes to see if they help your use case. I've created a PR at https://github.com/raspberrypi/libcamera-apps/pull/398 to use the alsa audio device directly instead of going through pulseaudio. This ought to reduce overhead slightly. You can select the alsa source by a new --audio-source alsa command line arugment.
Additionally, have you tried setting --audio-codec mp2 or some other codec from the default aac? It would be interesting to see if that helps as well.
Thanks, I'll try that asap!
I had tried misc formats a while ago, but time to try again with the recent bugfixes ^^
I've compiled this commit c3a9c97a0a84a1c20018e6ea5ed7027957f50690 (HEAD -> alsa, origin/alsa), and tested first on raspberry3B to check the absence of regressions.
I've stumbled upon a comeback of DTS errors, only for AVI files. It's weird because commits fixing DTS troubles are included in the commit tree. And generally my raspberrypi3B deals fine with combined recording, performance-wise (mpegts and mp4 formats have no such error so far, and no truncated audio track either).
Continuing tests, this time on raspberry pi zero.
pi@raspberrypi:~/attempts/libcamera-apps/build $ ./libcamera-vid -t 120000 --nopreview -o myfile3.avi --codec libav --libav-audio --libav-format=avi
[0:03:01.473298152] [1474] INFO Camera camera_manager.cpp:293 libcamera v0.0.1+21-7c855784
[0:03:01.571515941] [1475] INFO RPI raspberrypi.cpp:1414 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media2 and ISP device /dev/media0
[0:03:01.572712923] [1474] INFO Camera camera.cpp:1026 configuring streams: (0) 640x480-YUV420
[0:03:01.573308184] [1475] INFO RPI raspberrypi.cpp:800 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
[h264_v4l2m2m @ 0xdb3c80] <<< v4l2_encode_init: fmt=181/0
[h264_v4l2m2m @ 0xdb3c80] Using device /dev/video11
[h264_v4l2m2m @ 0xdb3c80] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0xdb3c80] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0xdb3c80] Failed to set number of B-frames: Invalid argument
[h264_v4l2m2m @ 0xdb3c80] Failed to set gop size: Invalid argument
Input #0, pulse, from 'default':
Duration: N/A, start: 1667209016.762929, bitrate: 1536 kb/s
Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Output #0, avi, to 'myfile3.avi':
Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 30 fps, 30 tbr, 30 tbn
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s
[avi @ 0xdb4ac0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 8 >= 8
terminate called after throwing an instance of 'std::runtime_error'
what(): libav: error writing output: -22
Aborted
pi@raspberrypi:~/attempts/libcamera-apps/build $ ./libcamera-vid -t 120000 --nopreview -o myfile3.avi --codec libav --libav-audio --libav-format=avi --audio-source alsa
[0:03:18.461144619] [1484] INFO Camera camera_manager.cpp:293 libcamera v0.0.1+21-7c855784
[0:03:18.532957775] [1485] INFO RPI raspberrypi.cpp:1414 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media2 and ISP device /dev/media0
[0:03:18.534038298] [1484] INFO Camera camera.cpp:1026 configuring streams: (0) 640x480-YUV420
[0:03:18.534700643] [1485] INFO RPI raspberrypi.cpp:800 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA
[h264_v4l2m2m @ 0xbffc80] <<< v4l2_encode_init: fmt=181/0
[h264_v4l2m2m @ 0xbffc80] Using device /dev/video11
[h264_v4l2m2m @ 0xbffc80] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode
[h264_v4l2m2m @ 0xbffc80] requesting formats: output=YU12 capture=H264
[h264_v4l2m2m @ 0xbffc80] Failed to set number of B-frames: Invalid argument
[h264_v4l2m2m @ 0xbffc80] Failed to set gop size: Invalid argument
Input #0, alsa, from 'default':
Duration: N/A, start: 1667209033.496256, bitrate: 1536 kb/s
Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s
Output #0, avi, to 'myfile3.avi':
Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 30 fps, 30 tbr, 30 tbn
Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s
[avi @ 0xc00060] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 8 >= 8
terminate called after throwing an instance of 'std::runtime_error'
what(): libav: error writing output: -22
Aborted
Hmm, that does seem a bit strange. Can you run libcamera-vid --version and report back the version/sha number generated?
Here it is :
pi@raspberrypi:~/attempts/libcamera-apps/build $ ./libcamera-vid --version libcamera-apps build: c3a9c97a0a84-intree 30-10-2022 (16:46:41) libcamera build: v0.0.1+21-7c855784
Thanks, that looks all correct to me. Do you still get the errors if you switch back to pulseaudio?
Here are the full tests on an old raspberry pi zero.
I have about the same issues with each specific format, be it with pulse (no --audio-source) or alsa source. Pulse is well the default, right ? But there are some changes in the process load, see comments at the end ^^
Mpegts would still be the most promising format, if that werent for the truncated audio track.
TESTS WITH DEFAULT AUDIO SOURCE
pi@raspberrypi:~/attempts/libcamera-apps/build $ ./libcamera-vid -t 120000 --nopreview -o pizerotests/myfile3.avi --codec libav --libav-audio --libav-format=avi [0:08:02.286671996] [1475] INFO Camera camera_manager.cpp:293 libcamera v0.0.1+21-7c855784 [0:08:02.979376879] [1476] INFO RPI raspberrypi.cpp:1414 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media2 and ISP device /dev/media0 [0:08:02.995332576] [1475] INFO Camera camera.cpp:1026 configuring streams: (0) 640x480-YUV420 [0:08:02.998580515] [1476] INFO RPI raspberrypi.cpp:800 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA [h264_v4l2m2m @ 0x332920] <<< v4l2_encode_init: fmt=181/0 [h264_v4l2m2m @ 0x332920] Using device /dev/video11 [h264_v4l2m2m @ 0x332920] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode [h264_v4l2m2m @ 0x332920] requesting formats: output=YU12 capture=H264 [h264_v4l2m2m @ 0x332920] Failed to set number of B-frames: Invalid argument [h264_v4l2m2m @ 0x332920] Failed to set gop size: Invalid argument Input #0, pulse, from 'default': Duration: N/A, start: 1667212254.354453, bitrate: 1536 kb/s Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s Output #0, avi, to 'pizerotests/myfile3.avi': Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 30 fps, 30 tbr, 30 tbn Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s [avi @ 0x3336f0] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 1648 >= 1648 terminate called after throwing an instance of 'std::runtime_error' what(): libav: error writing output: -22 Aborted
pi@raspberrypi:~/attempts/libcamera-apps/build $ ./libcamera-vid -t 120000 --nopreview -o pizerotests/myfile3.ts --codec libav --libav-audio --libav-format=mpegts [0:10:30.579390084] [1500] INFO Camera camera_manager.cpp:293 libcamera v0.0.1+21-7c855784 [0:10:30.900536106] [1501] INFO RPI raspberrypi.cpp:1414 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media2 and ISP device /dev/media0 [0:10:30.906873028] [1500] INFO Camera camera.cpp:1026 configuring streams: (0) 640x480-YUV420 [0:10:30.911589969] [1501] INFO RPI raspberrypi.cpp:800 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA [h264_v4l2m2m @ 0xaeb720] <<< v4l2_encode_init: fmt=181/0 [h264_v4l2m2m @ 0xaeb720] Using device /dev/video11 [h264_v4l2m2m @ 0xaeb720] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode [h264_v4l2m2m @ 0xaeb720] requesting formats: output=YU12 capture=H264 [h264_v4l2m2m @ 0xaeb720] Failed to set number of B-frames: Invalid argument [h264_v4l2m2m @ 0xaeb720] Failed to set gop size: Invalid argument Input #0, pulse, from 'default': Duration: N/A, start: 1667212401.286586, bitrate: 1536 kb/s Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s Output #0, mpegts, to 'pizerotests/myfile3.ts': Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 30 fps, 30 tbr, 1000k tbn Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s Halting: reached timeout of 120000 milliseconds. [aac @ 0xafd450] Qavg: 179.367
BUT ERROR WITH MPEGTS: TRACK TRUNCATED AT 40s
pi@raspberrypi:~/attempts/libcamera-apps/build $ ./libcamera-vid -t 120000 --nopreview -o pizerotests/myfile3.mp4 --codec libav --libav-audio --libav-format=mp4 [0:13:27.577060356] [1527] INFO Camera camera_manager.cpp:293 libcamera v0.0.1+21-7c855784 [0:13:27.893629227] [1528] INFO RPI raspberrypi.cpp:1414 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media2 and ISP device /dev/media0 [0:13:27.899655167] [1527] INFO Camera camera.cpp:1026 configuring streams: (0) 640x480-YUV420 [0:13:27.904481119] [1528] INFO RPI raspberrypi.cpp:800 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA [h264_v4l2m2m @ 0x1e1d920] <<< v4l2_encode_init: fmt=181/0 [h264_v4l2m2m @ 0x1e1d920] Using device /dev/video11 [h264_v4l2m2m @ 0x1e1d920] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode [h264_v4l2m2m @ 0x1e1d920] requesting formats: output=YU12 capture=H264 [h264_v4l2m2m @ 0x1e1d920] Failed to set number of B-frames: Invalid argument [h264_v4l2m2m @ 0x1e1d920] Failed to set gop size: Invalid argument Input #0, pulse, from 'default': Duration: N/A, start: 1667212578.272649, bitrate: 1536 kb/s Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s Output #0, mp4, to 'pizerotests/myfile3.mp4': Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 30 fps, 30 tbr, 1000k tbn Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s [h264_v4l2m2m @ 0x1e1d920] Pkt tracking failure: pts=57, track[57]=185 [mp4 @ 0x1e1e6f0] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly [mp4 @ 0x1e1e6f0] Encoder did not produce proper pts, making some up. [h264_v4l2m2m @ 0x1e1d920] Pkt tracking failure: pts=58, track[58]=186 [h264_v4l2m2m @ 0x1e1d920] Pkt tracking failure: pts=59, track[59]=187 [h264_v4l2m2m @ 0x1e1d920] Pkt tracking failure: pts=60, track[60]=188 [h264_v4l2m2m @ 0x1e1d920] Pkt tracking failure: pts=61, track[61]=189 [h264_v4l2m2m @ 0x1e1d920] Pkt tracking failure: pts=62, track[62]=190 [h264_v4l2m2m @ 0x1e1d920] Pkt tracking failure: pts=63, track[63]=191 [h264_v4l2m2m @ 0x1e1d920] Pkt tracking failure: pts=64, track[64]=192 ...
TESTS WITH "--audio-source alsa"
pi@raspberrypi:~/attempts/libcamera-apps/build $ ./libcamera-vid -t 120000 --nopreview -o pizerotests/myfile3.avi --codec libav --libav-audio --libav-format=avi --audio-source alsa [0:17:45.342702398] [1602] INFO Camera camera_manager.cpp:293 libcamera v0.0.1+21-7c855784 [0:17:45.702819541] [1603] INFO RPI raspberrypi.cpp:1414 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media2 and ISP device /dev/media0 [0:17:45.713172459] [1602] INFO Camera camera.cpp:1026 configuring streams: (0) 640x480-YUV420 [0:17:45.729286331] [1603] INFO RPI raspberrypi.cpp:800 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA [h264_v4l2m2m @ 0xa8b950] <<< v4l2_encode_init: fmt=181/0 [h264_v4l2m2m @ 0xa8b950] Using device /dev/video11 [h264_v4l2m2m @ 0xa8b950] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode [h264_v4l2m2m @ 0xa8b950] requesting formats: output=YU12 capture=H264 [h264_v4l2m2m @ 0xa8b950] Failed to set number of B-frames: Invalid argument [h264_v4l2m2m @ 0xa8b950] Failed to set gop size: Invalid argument Input #0, alsa, from 'default': Duration: N/A, start: 1667212836.517535, bitrate: 1536 kb/s Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s Output #0, avi, to 'pizerotests/myfile3.avi': Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 30 fps, 30 tbr, 30 tbn Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s [avi @ 0xa8bd00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 24 >= 24 terminate called after throwing an instance of 'std::runtime_error' what(): libav: error writing output: -22 Aborted
pi@raspberrypi:~/attempts/libcamera-apps/build $ ./libcamera-vid -t 120000 --nopreview -o pizerotests/myfile3-alsa.ts --codec libav --libav-audio --libav-format=mpegts --audio-source alsa [0:19:01.257964089] [1612] INFO Camera camera_manager.cpp:293 libcamera v0.0.1+21-7c855784 [0:19:01.573735875] [1613] INFO RPI raspberrypi.cpp:1414 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media2 and ISP device /dev/media0 [0:19:01.581765793] [1612] INFO Camera camera.cpp:1026 configuring streams: (0) 640x480-YUV420 [0:19:01.586024750] [1613] INFO RPI raspberrypi.cpp:800 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA [h264_v4l2m2m @ 0x1a5a910] <<< v4l2_encode_init: fmt=181/0 [h264_v4l2m2m @ 0x1a5a910] Using device /dev/video11 [h264_v4l2m2m @ 0x1a5a910] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode [h264_v4l2m2m @ 0x1a5a910] requesting formats: output=YU12 capture=H264 [h264_v4l2m2m @ 0x1a5a910] Failed to set number of B-frames: Invalid argument [h264_v4l2m2m @ 0x1a5a910] Failed to set gop size: Invalid argument Input #0, alsa, from 'default': Duration: N/A, start: 1667212912.051347, bitrate: 1536 kb/s Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s Output #0, mpegts, to 'pizerotests/myfile3-alsa.ts': Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 30 fps, 30 tbr, 1000k tbn Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s Halting: reached timeout of 120000 milliseconds. [aac @ 0x1a7e580] Qavg: 178.092
BUT ERROR WITH MPEGTS: TRACK TRUNCATED TOWARDS 40+s
pi@raspberrypi:~/attempts/libcamera-apps/build $ ./libcamera-vid -t 120000 --nopreview -o pizerotests/myfile3-alsa.mp4 --codec libav --libav-audio --libav-format=mp4 --audio-source alsa [0:34:52.713224429] [1646] INFO Camera camera_manager.cpp:293 libcamera v0.0.1+21-7c855784 [0:34:52.982058359] [1647] INFO RPI raspberrypi.cpp:1414 Registered camera /base/soc/i2c0mux/i2c@1/ov5647@36 to Unicam device /dev/media2 and ISP device /dev/media0 [0:34:52.987165301] [1646] INFO Camera camera.cpp:1026 configuring streams: (0) 640x480-YUV420 [0:34:52.993917224] [1647] INFO RPI raspberrypi.cpp:800 Sensor: /base/soc/i2c0mux/i2c@1/ov5647@36 - Selected sensor format: 640x480-SGBRG10_1X10 - Selected unicam format: 640x480-pGAA [h264_v4l2m2m @ 0x1692830] <<< v4l2_encode_init: fmt=181/0 [h264_v4l2m2m @ 0x1692830] Using device /dev/video11 [h264_v4l2m2m @ 0x1692830] driver 'bcm2835-codec' on card 'bcm2835-codec-encode' in mplane mode [h264_v4l2m2m @ 0x1692830] requesting formats: output=YU12 capture=H264 [h264_v4l2m2m @ 0x1692830] Failed to set number of B-frames: Invalid argument [h264_v4l2m2m @ 0x1692830] Failed to set gop size: Invalid argument Input #0, alsa, from 'default': Duration: N/A, start: 1667213863.424512, bitrate: 1536 kb/s Stream #0:0: Audio: pcm_s16le, 48000 Hz, 2 channels, s16, 1536 kb/s Output #0, mp4, to 'pizerotests/myfile3-alsa.mp4': Stream #0:0: Video: h264 (h264_v4l2m2m), drm_prime(tv, smpte170m/smpte170m/bt709), 640x480, q=-1--1, 200 kb/s, 30 fps, 30 tbr, 1000k tbn Stream #0:1: Audio: aac (LC), 48000 Hz, stereo, fltp, 32 kb/s [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=104, track[104]=232 [mp4 @ 0x1692c10] Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly [mp4 @ 0x1692c10] Encoder did not produce proper pts, making some up. [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=105, track[105]=233 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=106, track[106]=234 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=107, track[107]=235 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=108, track[108]=236 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=109, track[109]=237 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=110, track[110]=238 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=111, track[111]=239 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=116, track[116]=244 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=117, track[117]=245 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=118, track[118]=246 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=119, track[119]=247 [h264_v4l2m2m @ 0x1692830] Pkt tracking failure: pts=120, track[120]=248 ... [h264_v4l2m2m @ 0xd0f950] V4L2 capture poll unexpected timeout: events=0x145 ...
CPU LOAD ANALYSIS
With standard mode (via pulseaudio), the libcamera-vid app takes about 60% of the CPU, and pulseaudio daemon 20%.
With ALSA audio mode, the libcamera-vid app takes about 70% of the CPU, and pulseaudio daemon takes 10% (it's still activated by the recording).
So it's like the CPU load goes from one process to the other when switching mode.
./libcamera-vid -t 120000 --nopreview -o pizerotests/myfile5-alsa.ts --codec libav --libav-audio --libav-format=mpegts --audio-source alsa
@pakal, can you confirm that you are building the Release target of libcamera-apps? It does make a difference when you run with a debug vs release build.
Additionally, how are you measuring CPU load? When I used htop while doing a recording, my CPU was pegged at 100% - but I may have not done this correctly.
I've just followed these instructions - https://www.raspberrypi.com/documentation/computers/camera_software.html#building-libcamera-apps I don't know which flag would allow to switch between debug and release build :?
For the CPU load I used "top", from a second ssh connection preferably (having a GUI terminal would increase the load I guess?) ; it stayed below 80% almost all the time, but I'll have to try htop maybe
I've just followed these instructions - https://www.raspberrypi.com/documentation/computers/camera_software.html#building-libcamera-apps I don't know which flag would allow to switch between debug and release build :?
That sounds correct, by default it would make a release build. You can specify this on the command line by using cmake .. -DCMAKE_BUILD_TYPE=Debug or cmake .. -DCMAKE_BUILD_TYPE=Release
For the CPU load I used "top", from a second ssh connection preferably (having a GUI terminal would increase the load I guess?) ; it stayed below 80% almost all the time, but I'll have to try htop maybe
I would get top and htop will give identical results. Interesting that we see differences though. I will try with top as well.