blakeblackshear
High CPU Usage on cameras with a lot of motion
On dev as of this morning, CPU usage is much better for idle cameras. What surprises me is that on cameras with even a small amount of motion how much CPU usage goes up. It's not object detection - I have 4x corals, and they're all staying under 20ms inference speed (usually under 10ms). So my guess is it's either the object tracking code, or the motion region merging and prep-to-send-to-object-detection code. Given the recent mentions of avoid casts in numpy, I'm curious if similar optimizations are to be had in this section of the pipeline. See this py-spy top capture of a semi-active camera, run over 30-60s:
Besides the obligatory wait at the top, there are several methods using a significant chunk of CPU above improved_motion.py that look like internal numpy cast stuff.
to be clear, having corals does not mean the object detection is not using CPU, the regions still have to be resized to the model size and depending on what model you are running it may also need to have the pixel format changed as well.
Right. Indeed. And I'm not saying they're has to be optimizations here. It could be as good as it's going to get. I've just been impressed with how much @blakeblackshear improved the motion detector, and was hoping he could at least take a look at the other parts of process_frames after motion detection, but before the IPC to the object detector (but also including the create_tensor_input) for similar optimizations.
Full flame graph attached. Seems like a significant chunk is in match_and_update in norfair_tracker.py, particularly frigate_distance function.
I was just wanting to clarify that the assumption coral == no cpu usage for object detection isn't necessarily accurate, lots of motion will potentially lead to lots of regions and that can be part of the equation.
kindof off-topic but can we disable the "cpu is very slow 1077ms" message in the frigate ui (I know the cpu is very slow), it distracts from important messages like when a camera goes down.
With detection that slow your object detection is basically not functioning. There's a reason those messages are shown, we only show them when they indicate major problems.
With detection that slow your object detection is basically not functioning. There's a reason those messages are shown, we only show them when they indicate major problems.
I managed to get detection working very well during the day, around 90% accuracy, after properly setting up zones, motion masks, and object masks. Bounding boxes display correctly over people, and object tracking/lifespan features are functioning as expected.
I’m using the camera substreams for detections since the main 1080p stream produced black frames, but that’s not a big issue since substreams are typically used for this purpose anyway. Overall, I’m very satisfied with the results, especially across four cameras. CPU usage spikes to around 80% when an object snapshot is sent to the detector, but normally it stays in the idle range of 17–19% range when simply monitoring for motion.
From what I can tell, the system’s response time hovers near 900 ms (orange) and occasionally reaches 1000 ms (red). That’s still within an acceptable range — I haven’t noticed any negative impact on object detection performance, at least during daytime operation.
Your understanding is incorrect. At 1 second inference time you are going to have a high amount of skipped fps as even 5 detect fps for a single camera is not sustainable and frigate will skip frames and object tracking will not function well.
Your understanding is incorrect. At 1 second inference time you are going to have a high amount of skipped fps as even 5 detect fps for a single camera is not sustainable and frigate will skip frames and object tracking will not function well.
Sorry I mistyped, not 1000 ms, it's 100ms (0.1 second) to 200ms (0.2 second)