Feature Request: Bframe support

[vangourd]

It looks like all of my cameras produce B frames and even after disabling smart-codec and resorting to H264B. (basic).

I'm unable to prevent the following error for my Amcrest cameras which as far as I can tell causes issues with eventual web socket loss, lagging streams, and finally the early flushing of recordings resulting in 10 second duration captures.

20220702 22:45:46.740 s-driveway-main moonfire_nvr::streamer] driveway-main: sleeping for PT1S after error: pts not monotonically increasing; got 2232000 then 2232000

Backtrace:
   0: failure::backtrace::internal::InternalBacktrace::new
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.8/src/backtrace/internal.rs:46:44
   1: failure::backtrace::Backtrace::new
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.8/src/backtrace/mod.rs:121:35
      <failure::backtrace::Backtrace as core::default::Default>::default
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.8/src/backtrace/mod.rs:140:13
   2: <failure::error::error_impl::ErrorImpl as core::convert::From<F>>::from
             at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/failure-0.1.8/src/error/error_impl.rs:19:17
   3: moonfire_db::writer::Writer<C,D>::write
   4: moonfire_nvr::streamer::Streamer<C>::run_once
             at server/src/streamer.rs:266:13
      moonfire_nvr::streamer::Streamer<C>::run
             at server/src/streamer.rs:118:29
   5: moonfire_nvr::cmds::run::inner::{{closure}}::{{closure}}
             at server/src/cmds/run/mod.rs:349:25
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sys_common/backtrace.rs:122:18
   6: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/thread/mod.rs:498:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:492:40
      std::panicking::try
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panicking.rs:456:19
      std::panic::catch_unwind
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/panic.rs:137:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/thread/mod.rs:497:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/ops/function.rs:227:5
   7: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/boxed.rs:1861:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/boxed.rs:1861:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/std/src/sys/unix/thread.rs:108:17
   8: start_thread
   9: clone

This issue is the main problem with my common usage of moonfire-nvr so I'd like to document my issue and proceed as best as I can.

The Models of camera I'm using: IP8M-T2669EW-AI IP8M-2493EW IP2M-851EW

[vangourd]

Seems like the error is being thrown here. https://github.com/scottlamb/moonfire-nvr/blob/2e2de8cc6a618482e3527fb2602b05eb90547282/server/db/writer.rs#L774

Still trying to unpack what B-frame here is exactly doing. It seems B frame isn't common verbiage because I had some issues finding articles on it when I searched. Could just be I didn't know where to look.

I found an FFMPEG tutorial that has some background on it. http://dranger.com/ffmpeg/tutorial05.html

Suprisingly I found that tutorial code very similar to how the moonfire-nvr function is processing the PTS.

 let duration = pts_90k - unindexed.pts_90k;
            if duration <= 0 {

I still need to learn how Moonfire handles the PTS but according to that tutorial the receiver orders the frames to be played using the PTS. When B frames are involved they process information from the frame before and the frame ahead.

The only way I can see that being possible is if it's a static file, or if you await the current frame until the next frame comes in. That does seem fairly complex and I could see why for simplicity there isn't a current implementation.

Still looking into how I can help on this issue.

[scottlamb]

Are you cameras all Axis? Someone else just posted a similar issue on Retina (the RTSP library that Moonfire uses) saying newer Axis cameras use B-frames. I'd never heard of an IP camera doing this before.

B-frames I think are common in encoded TV shows/movies (Bluray, streaming services, etc.). I'm a little surprised to see IP cameras use B-frames because I think it means the cameras aren't configured for the lowest possible latency. As you've observed, Moonfire doesn't support B-frames today, but I'm totally open to changing that.

I think a good first step would be to make Retina ~~decode~~ [edit: used the wrong word in haste. "depayload", rather. Retina/Moonfire don't really decode video] the H.264 properly and have its .mp4 example work properly. That example code isn't used by Moonfire but it's relatively simple and I think would be a good test.

In Moonfire itself, the biggest problem is that the video index format needs to be changed to support them. So it will require a storage schema change.

[vangourd]

They are all Amcrest cameras. Yeah the concept seems weird to me but having worked in environments in the past with close to 1000 cameras I could see quite a bit of bandwidth savings from using B frames. As long as the guarantee is that the dependent frames come before the B frame you technically have all the info you need so it's basically compression logic.

As far as what I've read on it. I don't think latency is an issue as long as you're processing the frame before the next refresh cycle you're technically completing the decompression before the next frame. I THINK . My participation is really pushing my abilities and experience programming so please do not defer to me.

I also wonder if the issue here is really B frames. From what I read an identical PTS doesn't make sense for a B frame. It still should have a unique PTS because it represents a distinct frame it just has a "retro" PTS because the prior frames DTS had to be lower so you have all the data to build the "B" frame for presentation. So that would mean the difference would be negative? I still need to understand better how Moonfire is scheduling frames. The file this error is coming from is the writer so "scheduling" here is just ordering them to be written to the sample file I think.

[vangourd]

I notice on the other similar issue you linked. It's actually showing a negative -5400 PTS difference which makes sense.

The B frame is put out of order after its dependent frames and therefore has a negative PTS value.

What's weird to me is why I'm seeing duplicate PTS values. Why would we try to display two frames at the same time? Seems like it's a different issue and would explain why my cheap Amcrest cameras with which some are getting older have a feature only seen on Axis cameras (primo).

[scottlamb]

I haven't studied how B frame pts/dts are supposed to be represented on the wire, but the authoritative reference should be this one: https://datatracker.ietf.org/doc/html/rfc6184

[scottlamb]

...and the main H.264 spec (although it's a dense, difficult read!). Links at https://github.com/scottlamb/moonfire-nvr/wiki/Standards-and-specifications

[vangourd]

Alright this is perfect. When I get some free time I'll get a pcap from tcpdump and see what the packets show.

[scottlamb]

Any chance you could get that pcap? I'd like to see how this camera's sending it on the wire still.

fwiw, I ran across the gstreamer 1.22 release notes, which include this:

Muxers are often picky and need proper PTS/DTS timestamps set on the input buffers, but that can be a problem if the encoded input media stream comes from a source that doesn't provide proper signalling of DTS, such as is often the case for RTP, RTSP and WebRTC streams or Matroska container files. Theoretically parsers should be able to fix this up, but it would probably require fairly invasive changes in the parsers, so two new elements h264timestamper and h265timestamper bridge the gap in the meantime and can reconstruct missing PTS/DTS.

Possibly the right thing to do is to copy their approach.

[vangourd]

No sorry ended up switching jobs and been having to table my side projects a bit.

I'm still dabbling in Rust but trying to up my skills so I can be of more help.

I have a cluster setup though I could try a specific version again and try to get that PCAP.

Just remind me what you're wanting to see again.

Basically the cut out problem with the B frames again?

On Wed, Feb 15, 2023, 8:32 PM 'Scott Lamb' via Applications < @.***> wrote:

Any chance you could get that pcap? I'd like to see how this camera's sending it on the wire still.

fwiw, I ran across the gstreamer 1.22 release notes https://gstreamer.freedesktop.org/releases/1.22/, which include this:

Muxers are often picky and need proper PTS/DTS timestamps set on the input buffers, but that can be a problem if the encoded input media stream comes from a source that doesn't provide proper signalling of DTS, such as is often the case for RTP, RTSP and WebRTC streams or Matroska container files. Theoretically parsers should be able to fix this up, but it would probably require fairly invasive changes in the parsers, so two new elements h264timestamper and h265timestamper bridge the gap in the meantime and can reconstruct missing PTS/DTS.

Possibly the right thing to do is to copy their approach.

— Reply to this email directly, view it on GitHub https://github.com/scottlamb/moonfire-nvr/issues/231#issuecomment-1432334144, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFI7ZUP74QESLQRHF4BYU5TWXV7RHANCNFSM52PXF7TA . You are receiving this because you authored the thread.Message ID: @.***>

[scottlamb]

No worries. I'd like to see a pcap of a RTSP stream including B-frames. The cut-out problem with Moonfire would be helpful; a stream from another RTSP client that doesn't cut out would probably be even better.

[vangourd]

Is https://hub.docker.com/r/scottlamb/moonfire-nvr still up to date?

I can test this out but would prefer to run it locally on Kubernetes if I can.

On Thu, Feb 16, 2023 at 1:41 AM 'Scott Lamb' via Applications < @.***> wrote:

No worries. I'd like to see a pcap of a RTSP stream including B-frames. The cut-out problem with Moonfire would be helpful; a stream from another RTSP client that doesn't cut out would probably be even better.

— Reply to this email directly, view it on GitHub https://github.com/scottlamb/moonfire-nvr/issues/231#issuecomment-1432600756, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFI7ZUJHYIZKYHNZLPW66S3WXXDZ3ANCNFSM52PXF7TA . You are receiving this because you authored the thread.Message ID: @.***>