Linked monitors not recording when parent goes into alert

[maddios]

Describe Your Environment

• Version of ZoneMinder: 1.36.10
• How you installed ZoneMinder: from-source
• Full name and version of OS: Ubuntu 18.04 LTS
• Server is a 16 core 2.4ghz xeon with 64GB ram, my whole setup runs at about 10 load average while using approx 30GB ram.

I'm running 5 Amcrest IP cameras each outputting 2 streams (low and high rez), I'm actually doing my triggering in reverse though, I have my high res streams running MoDect and the low res streams actually go to a ML object detection/motion tracking software I wrote which then outputs a stream which I'm also streaming into zoneminder, These 5 AI streams are as NoDect. Each of these 5 streams has its full res stream set as a Linked monitor.

Describe the bug The issue I'm having is that MOST of the time my linked monitors don't record anything, I've tried restarting ZM and triggering an alert once ZM is back up and it'll sometimes trigger and sometimes not. As far as I can tell there's nothing in the log files to indicate any kind of issue even with debug mode enabled. I've check the zmc_m{id}.log files and the zmwatch.log files.

To Reproduce I have no idea how to reproduce this other than it happens in my env all the time.

Expected behavior I expect that every single time an alert is triggered all linked cameras must record it

Debug Logs parent linked

first log shows an alert around 19:14:59.339395, but the 2nd one never does

[welcome[bot]]

Thanks for opening your first issue here! Just a reminder, this forum is for Bug Reports only. Be sure to follow the issue template!

[maddios]

Actually, I forgot to mention that the triggering monitor has a total of 4 linked monitors, 3 other cameras link to it as well as the AI stream. Not sure if this causes some sort of mutex issue

[connortechnology]

Whoops, accidentally closed this with an unrelated commit.

Um there are no mutexes involved. we will need debug level 3 logs from all related zmc processes.

[maddios]

the logs I included were level 9, is level 3 more verbose? or does level 3 specifically log the mutexes?

[SteveGilvarry]

Only got INF records in the logs you posted, should see DBG lines in debug logs. Also normally set to log to separate files for each process. Level 3 is lower and less verbose than 9. So you might need it for both monitors using _zmc_mX options

This might help or just ask if no luck. https://zoneminder.readthedocs.io/en/stable/userguide/options/options_logging.html

[maddios]

Wait where are you seeing INF records in these, these are the zm_debug.log.{id} files. When I download the 2 files I just get a bunch of stuff like this:

12/01/21 19:14:49.607387 zmc_m2[37395].DB4-zm_packetqueue.cpp/431 [Locking in get_packet using it 0x560ef41db6c0 queue end? 0]
12/01/21 19:14:49.607392 zmc_m2[37395].DB4-zm_packetqueue.cpp/436 [Have Lock in get_packet]
12/01/21 19:14:49.607397 zmc_m2[37395].DB3-zm_packetqueue.cpp/450 [get_packet using it 0x560ef41db6c0 locking index 33]
12/01/21 19:14:49.607402 zmc_m2[37395].DB4-zm_packet.h/106 [TryLocking packet 33]

But sure, I'll get a level 3 debug snapshot when I get a chance, i can see how level 9 might be too verbose.

[SteveGilvarry]

Sorry couldn't see the files from mobile app for GitHub. Was talking about the Debug logs you pasted into issue log.

[maddios]

Here are new zm debug logs for the 2 monitors with debug level 3 enabled:

zm_debug_parent.log zm_debug_linked.log

This was a Forced trigger and the linked monitor didn't record anything. Interestingly enough the other 3 linked monitors DID record this forced event BUT not the subsequent 5 which have happened since this forced event. Seems like the linked recordings DO work but VERY sporadically, which makes it so I'm forced to run my MoDect on my full resolution streams since I can't trust linked monitors to actually capture

[maddios]

is this a relevant thing?

12/10/21 20:37:17.906460 zmc_m2[5368].DB1-zm_monitor.cpp/2336 [Reloading linked monitors for monitor front door, '(null)']

shouldn't null here be the 4 linked monitors?

[connortechnology]

So the issue here is that the analysis thread was waiting on capture instead of looking for a linked monitor to be triggered. Capture took .2 of a second which is a long time actually. So linked monitor checking needs to be it's own thread... I can't imagine making this kind of change in 1.36. Maybe with enough testing.
That being said... the linked monitor DID notice the alarmed state of the primary... so I'm gonna have to look closer at the code. Keep to that code says I, Yeargh!

[maddios]

Thanks, I'm glad you can see the issue, it's ok if you can't do it in .36. I've been living with this "bug" for years, I just never really invested a ton of time into debugging it until now. Hopefully it can be addressed at some point in the future.

If you do release a fix I'll gladly test it.

[connortechnology]

I think I can improve it a lot. Ultimate fix in a few months. At the end of the day we will always be polling and will have to sleep unless we consume 100% cpu.
I think long term we will use mqtt to queue a notice about the alarm status and so even though we might notice it late, we will still get it.

[maddios]

yeah, even if it triggers within a second or two it should still be fine, at least for my setup. Can always increase the pre-alarm frame count.