frigate
frigate copied to clipboard
blakeblackshear
Audio Classification via Tensorflow Lite
Describe what you are trying to accomplish and why in non technical terms
Instead of throwing away the audio that is captured by Frigate, it would be cool to be able to use similar ML detection tricks on the audio to build binary sensors to detect various sounds heard by the cameras
Describe the solution you'd like
Similar to ImageNet, YamNet is a free implementation of recognizing a bunch of different audio samples. Here's the total list of sounds that it detects.
Some interesting ones:
- Speech / voices
- Vehicle / Car
- Various Animals (Bird/Cat/Dog)
- The sound of something dropping or falling
- Sirens and alarms (a similar feature exists on many commercial camera platforms)
- Emergency Vehicle
- Finger Snapping / Clapping, reimplement the most extra Clapper you've ever seen 😅
Describe alternatives you've considered
This could be built standalone but it would also require the Edge TPU to be performant and I suspect there is no resource-sharing between Edge TPUs (i.e. you open it for exclusive access), as well as requiring extra ffmpeg processes => extra CPU usage. This would also involve recreating the infrastructure to talk to Home Assistant as well
maybe should be designed as a second AI-level? means "chaining" AI? like its discussed for utilitize also other AI-models tor object classification in frigate?
@ozett I don't think that these should be chained in most cases, but it definitely might be related to #1697 in terms of the infrastructure
baby crying! it is must to have, it will simply add electronic nanny functionality out of box
Adding on that this feature would ideally be a foundation to support speech transcription and voice fingerprint, in addition to sound labeling.
Sound labels should include gunshot
I would love to see this, and while I don't have any experience coding in frigate (I just started playing with it), I would love to help if someone can point me in the right direction
I would love to see this, and while I don't have any experience coding in frigate (I just started playing with it), I would love to help if someone can point me in the right direction
https://docs.frigate.video/contributing
There's a lot to be explored for this feature, PRs are appreciated
I'd gladly place a bounty of $500 for someone to implement this, would love to use Frigate as a baby monitor.
