frigate
frigate copied to clipboard
blakeblackshear
Semantic Search for Events
Implementation for #8980
Still a work in progress, but far enough along to play around with if interested. I still need to implement a way to keep the Events database and the vector database in sync and add missing historical events to the vector database on startup.
And yes, I know that I added the frontend to the old web - but the new frontend only uses timeline for now, and it does not have enough detail to perform this type of search, or filters implemented at this point in time.
Implementation is mostly done at this point. I implemented the frontend in web-old because it was further along (and less work) than trying something in the new UI.
For the new UI, what do you think about a search bar at the top that is available on every page? When searched, it shows a styled dropdown of matching events allowing for additional filtering, downloading, or opening in the recordings/timeline viewer.
Examples
Search for "man with brown hair and beard"
Most results from description
Search for "golfers"
Most results from description
Search for "black and white dog"
Clicked "Find Similar" and returned image search results
I just installed this on my parent's instance without configuring Gemini. Just using the CLIP model gets some pretty decent results:
Example 1
Search Term: "chevy bolt"
Example 2:
Search Term: "package delivery"
Filtered to the day of 12/15
Requested similar images to the first result shown in the screenshot
I've updated the docs and have been running this in production for a couple of days. I think it's ready for review now.
you seen this project?
I had not until now. GPT-4 Vision can be integrated this way relatively easy as well, or that project can be updated to submit the summary as a description directly to Frigate and have it indexed and searchable.
I like the use of multiple image frames representing a fuller context, but wanted to keep it relatively simple for the first implementation.
