[RFC] Trigger Clips and Snapshots via API

[herostrat]

Idea:

Allow triggering of clips and snapshots via external command.

Implementation:

Implement API endpoints for:

• Webinterface
  • In camera view two buttons
• HTTP
  • /api/<camera_name>/create-snapshot
  • /api/<camera_name>/create-clip
• MQTT
  • frigate/<camera_name>/snapshots/create
  • frigate/<camera_name>/clips/create

The configuration of the created object is according to the user config and therefore the same as triggered via internal detection.

Background/Usecases:

• motion/PIR/light barrier sensors in home automation as trigger condition
• timer based trigger condition
• ...

@blakeblackshear is this acceptable or according to the projects vision? I am interested in implementing this myself, so it is no FR

@all Additional ideas and suggestions?

---

Relevant Issues/Discussions: https://github.com/blakeblackshear/frigate/issues/1040 https://github.com/blakeblackshear/frigate/issues/926 https://github.com/blakeblackshear/frigate/issues/345 https://github.com/blakeblackshear/frigate/discussions/1008

[blakeblackshear]

I am not completely convinced a few other enhancements wouldn't obviate the need for this feature. What if the 24/7 recordings could be set to only retain segments that had motion? Also, what if 24/7 recordings could be turned on and off like detection, clips, and snapshots? I don't see any use cases in the linked tickets that wouldn't be covered by these two enhancements.

If this is implemented, my gut says this should be done by manually saving a frame as a snapshot and putting an event into the event queue for processing. I think it will need to have a predetermined length, and the properties on the event table will need to be reconciled to accommodate a manual recording. It should fit fairly easily into the existing event handling and clip creation.

[herostrat]

I am not sure I understand correctly and/or made it clear enough what my idea is.

Basically I want to use the event queue, but add additional triggers for it.

My understanding is that currently the only way to add something to.the event queue is when frigate has detected something.

Please correct me if I am wrong, I am most certaintly not too familiar with the codebase.

How would you use external sensors to generate clips oe snapshots and save them?

[blakeblackshear]

I understand the idea and agree with the implementation of using the event queue. That's basically what I said.

I just don't see the need to use external sensors to trigger clips or snapshots if Frigate is working properly. I would rather ensure Frigate has the features to capture anything of interest than rely on external systems to turn things on and off.

[herostrat]

Ah I see, sorry for the confusion. English is not my mother tongue.

I agree for the most part. In my mind there are a few usecases where frigate is working and detecting properly, but also could use this feature. Sometimes there is just nothing to detect in the video feed but I want a clip nonetheless.

Thanks for your feedback!

[Whytey]

Additional use case - video doorbell. Doorbell is pressed so trigger a clip/recording. Frigate might not have detected motion (maybe the local kids are being tricky and hiding themselves??) but you would like to have a clip so you can see what was going on in the background??

[blakeblackshear]

If they were hiding themselves and frigate doesn't detect any motion, wouldn't you just be recording a static image?

[Whytey]

I was thinking more along the lines of them being in the distance but too small to be within the detection threshold. Corner cases admittedly.

[blakeblackshear]

And those far away objects are related to your doorbell ringing? I'm trying to understand if this is a hypothetical scenario or if you have a use case I haven't considered.

[Whytey]

Purely hypothetical.

I like the idea of externally triggering Frigate... Though purely due to having had poor experiences of false positives with other NVR solutions.

[stephengolub]

I like the idea of triggering a recording with "custom events" and was actually looking into the code to see how complicated it would be. I ran out of free time a while back, so maybe I'll revisit it.

Here's where I think just relying on motion/object detection could fail: Say you only have person detection set up, and a delivery driver comes up with a load of packages or big jacket or something and the model just doesn't quite recognize them. Then you miss the recording.

Having the "custom event" (in this case a "ding dong" or "ring" event) that you can send a payload to Frigate and say:

{
  "camera": "front_door",
  "event": "ring"
}

I can also imagine triggering perimeter recording if my door bell rings, or almost any other type of thing.

As an MVP, I'd just accept event and camera and use the default clip recordings. Maybe in the future add the ability to specify length, etc.

[blakeblackshear]

If 24/7 recordings had the ability to only retain segments where motion was detected, wouldn't you cover the edge cases where the model didn't detect for some reason?

Also seems like a switch to turn off 24/7 recordings would address the use case of turning on recording based on an event.

[stephengolub]

My thinking is that it would tag a point in time by leveraging the existing concept of "events".

By adding a custom "ding" event or whatever, I will see those separated out in the UI and database and could even have a "last_ding" camera in home assistant.

Another use case I would use was "garage" event. So if someone opens or closes my garage, I would run a snapshot of the driveway camera as well and have the specific point in time tagged.

[blakeblackshear]

Those seem like worthwhile use cases. This should fit reasonably well into the the existing architecture as I mentioned above.

[tam-wh]

Ideally, I would use object detection in frigate but I don't have a coral and I rather not max up my CPU for object detection. Since I have a PIR sensor in home assistant, it would be great to use that to trigger an event in frigate and record a clip. (E.g. If someone enters my room when I'm away)

Ability to control 24/7 recording with Mqtt/hass is important as well for certain use case (e.g. using hass automation to stop recording when I'm home and start recording when I'm away)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[blakeblackshear]

@herostrat please don't spend any time on this. Upcoming changes will make manually triggered events much easier to implement, so any work here would be a waste of time.

[vhbramos]

I have a few use cases that could leverage this feature:

1. Audio Detection: My Amcrest cameras can detect audio and it would be great to manually trigger a clip recording in Frigate. There is no motion, but the camera audio can provide important insights about ongoing events. Example: Detection of fire/smoke alarms or any other audio alarms. ( Wize cameras have a specific event to alert users of Fire/Smoke alarms )

2. Tripwire: This can be a Frigate core feature in the future, but in the meantime I could use an external sensor ( some cameras also have it embedded ) to trigger a recording and a specific alert if motion is detected from a specific area to a specific area. This would be an additional alert, since I would still want Frigate to monitor a larger area for objects ( like people and cars )

3. Other AI detection: I am looking to deploy a couple of cameras with more advanced AI capabilities, like detecting parked cars and objects left in specific areas. I can get motion events for those situations in Frigate today but those extra labels would help me get better notifications

[zaPure]

I agree that this would be a useful feature to trigger recording from MQTT events, e.g. my alarm or electrical fence triggers or garden beams go off. I would like the camera to start recording so that I can get a sense of what triggered it. As I have my cameras are set up only to record on people detection.

[oramirite]

I'd like to chine in on the need for this as well. I regards to Frigate working properly obfuscating the need for this feaure: I feel it's anything but. Frigate itself, when working correctly, can benefit from it's own detection to trigger events manually.

I have an overhead street camera that's at a perfect angle for person detection. Frigate detects people on my stoop and even near my mailbox perfectly. My doorbell camera points out at a busy street however, and unless I can get some kind of z-depth detection it's impossible to mask things out in a way that will only detect people on my stoop. The overhead camera is perfect for drawing zones. But the doorbell camera is the only one that can capture faces. Plus, it captures audio. There is simply no way for that camera on it's own to properly filter out noise though. It really needs the benefit of other sensors to retain events properly.

It would be great to rely on the overhead camera's detection, and be able to roll my own logic for capturing the same clips on my doorbell camera. This way, the media library of my Doorbell would only ever contain events where people were accurately detected on my stoop.

If I enable detection right now, I'm sure the doorbell will capture that same clip of the person on my stoop... along with at least 200+ other people that day who have simply passed by. The doorbell camera angle is just REALLY bad for actual detection, but works perfect for faces and general viewing when the clips are captured. The difference is less than a pixel from the doorbell angle whereas the overhead angle makes it simple.

While I appreciate that Frigate wants to cover and and all use-cases through detection, I feel like there's always going to be edge cases where complimenting the detection with an additional sensor will work perfectly.

[blakeblackshear]

You should create a zone for your stoop and add that zone to required zones for your doorbell camera. Then you will only get events where the person stepped onto your stoop.

Either way, this feature is coming. No need to continue to advocate for it.

[oramirite]

That's great news @blakeblackshear, thanks so much!! And thanks for such a quick reply before.

I can't see the ground due to this doorbell camera's angle, so there's nowhere to draw that zone you mentioned. The overhead camera I mentioned works flawlessly with zones so using that as a trigger will be perfect. It's no fault of Frigate at all - my house is elevated from the street, so the doorbell camera only sees humans from the waist up. Detection works great... it's just not possible to differentiate someone walking past my house from them being at the door.

Excited for this addition. Thank you so much for creating what might be the best NVR solution out there!!

[Notrial]

Has this feature been implemented in the recent version updates?

[herostrat]

Unfortunately not, I tried to add it but was not able to integrate it properly into the event queue.

I will probably not have time to continue for a bit, but if someone finds a easy way to implement it feel free!

[Notrial]

That's really too bad...having on-demand clips would be really useful. Are there any workarounds to getting screenshots/clips from Home Assistant on button press for example?

[herostrat]

afaik not

[oramirite]

Dang, that's really disappointing. I appreciate the attempt. Sounds like this is no longer on the official roadmap though?

I don't mean to be the peanut gallery here, but the solutions being discussed feel really overcomplicated. Integration into the event queue would be nice, but it sounds like an uphill battle since you then need to re-create the conditions of a detection. We really just want to manually trigger a clip recording. Simple. It should be as un-related to motion detection as possible.

@blakeblackshear you mentioned this as well: "I just don't see the need to use external sensors to trigger clips or snapshots if Frigate is working properly. I would rather ensure Frigate has the features to capture anything of interest than rely on external systems to turn things on and off."

While I appreciate that Frigate really wants to lean in to AI detection, you're always going to have situations that won't cover, even when it's working perfectly. Also, Frigate is built to integrate extremely closely with Home Assistant - a platform for automation and sensor data. The strength of this integration very much SHOULD be input from HA. I don't really understand the logic that Frigate shouldn't allow external input.

Frigate isn't failing by needing manual intervention. It is, in fact, a testament to how WELL the AI detection is implemented that we're already seeing it's limits. There will always be situations where AI + motion detection isn't the whole picture, and we really need the tools to fill in that last 5% of situations.

[blakeblackshear]

@oramirite As I stated a few comments up.

Either way, this feature is coming. No need to continue to advocate for it.

[herostrat]

Yes, sorry if I gave you the impression it is not or no longer planned.

I just meant I was not able to add it in a satisfactory way. This does not mean it is not possible, just that I was not able to within the time I had available.

[NickM-27]

@Notrial I have been able to implement a lite version of this on the HASS side. When someone presses my doorbell I send a notification to my TV / Mobile Devices using the http://frigate/api/<camera_name>/latest.jpg as the image source. Could also use that to save an image.

As stated this is coming to frigate natively, but thought I would share for anyone trying to create something similar to my use case.

[raintonr]

When someone presses my doorbell I send a notification to my TV / Mobile Devices using the http://frigate/api/<camera_name>/latest.jpg as the image source.

In fact this is exactly how a Loxone intercom block functions. It works very well but guess it would be nice to see all events of all types in the same UI.