blakeblackshear
Detect change in scene after event, e.g. added or removed items
It would be cool to have Frigate be able to detect/alert on significant changes in a scene from 'before' to 'after' a detected event. This would be handy for detecting situations where something visible was done, or an object was left or taken, and it's not clear when it occurred. Some commercial NVRs claim in their ad copy that they can do this.
Example: A camera has a view of an area with moderate foot traffic. Someone enters the zone, drops a bag of garbage and leaves. Or: an unclassified object, say a suitcase, is in view. Someone enters the zone, removes it and leaves the zone.
Not being an expert in image processing, I'm not sure how this would be implemented.
I'd probably try taking a snapshot from the beginning and end of the event, and comparing them somehow. And then I'd be disappointed by how poorly this works if there's more than one moving object in the image, so I'd take like a second of video from the beginning and end, and create a distilled image of each consisting only of the pixels that did not change significantly during the clip. Not sure what sort of comparison would work best, probably not just raw pixel values due to lighting changes etc. Color change? Edge detection? Something that can piggyback on the existing moving/stationary object detection?
This seems like something that GenAI would handle, not sure how dahua implements it but in general it seems like it would be prone to false positives. What if I'm just taking out the trash or moving it back, I don't want to have a more advanced alert for that.
I'd see it more as a sorting and narrowing tool: during the course of the day, 50 people were detected passing through the zone, but only one of them stole the generator that was sitting there. Perhaps think of it as a flag attached to an existing detection or alert, saying "this event changed the scene noticeably." Even in the garbage scenario, it might be handy. Suppose a truck drove by and stole your garbage can, and you wanted to pick out that event from any other "car drove by" events.
I suspect it would be hard to do this well, or perhaps at all. But it sure would be neat.
I've done a little poking at an Axis system I admin, and it seems it might be doing approximately this: Take a snapshot every time a zone transitions from "active/moving objects" to "no active/moving objects", compare those snapshots and attribute any changes to the last detected event in the zone.
Maybe could be done passing both images to an image capable LLM and asking it to give you the differences, but anything could come out of it, not sure hot to limit it, maybe with a good system prompt that tells the LLM to only respond with noticeable changes like "objects that had disappeared, or garbage bags added to the scene and ignore the position of the dog" for example.
A more classical approach to solve the same problem would be some sort of "bisect" system where you start from a known date where the object was there, (or the thing hadn't happened, etc...) and frigate would display snapshots that progressively narrow down the time range, and the user would then select if whatever they were looking for was present.
The bisect approach is more or less what I use now, but manually by clicking through the timeline. Probably wouldn't be terribly difficult to automate that.
Image capable LLM might be a bit much, probably requiring either cloud service or pretty beefy hardware to run such an LLM locally.
If my workload clears up a little bit, this is on my agenda to explore, maybe just some shoddy python opencv stuff to see if I can find a semi-reliable image comparator.
Maybe a combination approach, you provide Frigate a positive and negative snapshot, as well as region of interest and then Frigate uses a vision LLM to do the bisect automatically?
As far as I konw, Dahua's NVR and cams can do this well, it can detect the object were left behind or be taken. It can also set the left time threshold as you need. It's basicly use the object track_id which can consider that the object is here for all the time or there's a new object in the screen. I use this in my job by self-train model with yolo and it done pretty well for any moving or static vehicle and person. If Frigate support using mutiple model for each cam, I think it's an easy way to achieve this.
