frigate
frigate copied to clipboard
[Detector Support]: Doesn't work GPU with OpenVINO
Describe the problem you are having
When I am trying to enable OpenVINO detector with device=GPU I can see the error in log and Frigate becomes inactive for a while. If I set device=AUTO it starts to work, but when detect is enabled I can see high cpu utilization and no any activity in intel_gpu_top which shows that GPU acceleration is not working. HW acceleration of camera is working good.
I set LIBVA_DRIVER_NAME=iHD because of if I choose i965 then I see how the blitter in intel_gpu_top is connected, I'm not sure, but it doesn't seem good and openvino doesn't work also.
I have Celeron J4105 and Unraid system.
Version
0.13
Frigate config file
database:
path: /config/frigate.db
mqtt:
enabled: true
host: homeassistant.local
port: 1883
user: admin
password: password
topic_prefix: frigate
client_id: frigate
ui:
live_mode: mse
use_experimental: true
birdseye:
enabled: false
detect:
enabled: true
detectors:
ov:
type: openvino
device: GPU
model:
path: /openvino-model/ssdlite_mobilenet_v2.xml
model:
width: 300
height: 300
input_tensor: nhwc
input_pixel_format: bgr
labelmap_path: /openvino-model/coco_91cl_bkgr.txt
snapshots:
enabled: true
record:
enabled: true
expire_interval: 60
retain:
days: 12
mode: all
events:
retain:
default: 12
mode: motion
ffmpeg:
hwaccel_args: preset-vaapi
logger:
default: error
logs:
frigate.mqtt: error
frigate.app: error
frigate.ffmpeg: critical
go2rtc:
log:
format: text
level: error
# api: trace
# exec: debug
# ngrok: info
# rtsp: warn
# streams: error
# webrtc: fatal
rtsp:
default_query: mp4
streams:
cam3:
- rtsp://admin:[email protected]:554
cam3_sub:
- rtsp://admin:[email protected]:554/Streaming/channels/2
cameras:
cam3:
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/cam3
input_args: preset-rtsp-restream-low-latency
roles:
- record
- path: rtsp://127.0.0.1:8554/cam3_sub
input_args: preset-rtsp-restream
roles:
- detect
output_args:
record: preset-record-generic-audio-copy
motion:
mask:
- 0,176,110,176,110,155,0,155
docker-compose file or Docker CLI command
docker run
-d
--name='frigate'
--net='bridge'
--privileged=true
-e TZ="Europe/Moscow"
-e HOST_OS="Unraid"
-e HOST_HOSTNAME="SERVER"
-e HOST_CONTAINERNAME="frigate"
-e 'FRIGATE_RTSP_PASSWORD'='password'
-e 'LIBVA_DRIVER_NAME'='iHD'
-l net.unraid.docker.managed=dockerman
-l net.unraid.docker.webui='http://[IP]:[PORT:5000]'
-l net.unraid.docker.icon='https://raw.githubusercontent.com/yayitazale/unraid-templates/main/frigate.png'
-p '5000:5000/tcp'
-p '8554:8554/tcp'
-p '8555:8555/tcp'
-p '8555:8555/udp'
-p '1984:1984/tcp'
-v '/mnt/user/appdata/frigate':'/config':'rw'
-v '/mnt/disks/cameras/':'/media/frigate':'rw,slave'
-v '/etc/localtime':'/etc/localtime':'rw'
--device='/dev/dri'
--shm-size=256mb
--mount type=tmpfs,target=/tmp/cache,tmpfs-size=2726297600
--restart unless-stopped 'ghcr.io/blakeblackshear/frigate:stable'
Relevant log output
2024-02-01 19:33:04.112890288 [INFO] Preparing Frigate...
2024-02-01 19:33:04.140557029 [INFO] Starting Frigate...
2024-02-01 19:33:06.708554578 [2024-02-01 19:33:06] frigate.app INFO : Starting Frigate (0.13.0-01e2d20)
2024-02-01 19:33:10.350863638 [2024-02-01 19:33:10] frigate.config WARNING : Customizing more than a detector model path is unsupported.
2024-02-01 19:33:12.743140257 Process detector:ov:
2024-02-01 19:33:12.746019189 Traceback (most recent call last):
2024-02-01 19:33:12.746050491 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-02-01 19:33:12.746053039 self.run()
2024-02-01 19:33:12.746055072 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-02-01 19:33:12.746061394 self._target(*self._args, **self._kwargs)
2024-02-01 19:33:12.746064435 File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2024-02-01 19:33:12.746091310 object_detector = LocalObjectDetector(detector_config=detector_config)
2024-02-01 19:33:12.746111650 File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
2024-02-01 19:33:12.746114358 self.detect_api = create_detector(detector_config)
2024-02-01 19:33:12.746116276 File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-02-01 19:33:12.746117776 return api(detector_config)
2024-02-01 19:33:12.746119702 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 32, in __init__
2024-02-01 19:33:12.746121416 self.interpreter = self.ov_core.compile_model(
2024-02-01 19:33:12.746123545 File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 399, in compile_model
2024-02-01 19:33:12.746126909 super().compile_model(model, device_name, {} if config is None else config),
2024-02-01 19:33:12.746181038 RuntimeError: cldnn program build failed! [GPU] clWaitForEvents, error code: -14
Operating system
UNRAID
Install method
Docker Compose
Coral version
CPU (no coral)
Any other information that may be helpful
No response
@NateMeyer have you seen this before?
No, I just try to setup it. Actually no, it seems I tried a month ago to setup OpenVINO when Frigate has ver 0,12 and it was the same.
This is I can see when set device=AUTO, but as I mentioned earlier, it doesn't work when detect is enabled.
to be clear I tagged someone else with that question
No that is a new one to me. I'll see what I can dig up later today.
It looks like the container is missing something like this one
That page is outdated. Also, many users are using this in GPU mode so it will likely be something host specific
Clear. Let me please know if I need to share more information about the case.
I just wanted to add that I found a message in syslog when I am trying to launch a Frigate with Openvino and device GPU
it seems that the error occurs that there is some OpenCL bag in kernel because of there are similar messages on the Internet related to the incorrect work of hardware acceleration for example in Plex. Hopefully, a solution to this problem will be found.
I have the exact same issue, same cpu in a nuc but with proxmox.
Indeed Plex struggled for a long time with some of these processors after an update but was resolved a few weeks back. As far as i know their issue was related to changes in the drivers used.
Same issue on a docker nested container inside a proxmox unprivileged container using intel N3350. Does the container and host both need some packages ? if so then which are needed ?
I seem to have the very same issue, is there any solution already?
I seem to have the very same issue, is there any solution already?
No, I didn't find and just bought Google Coral TPU
I have intel J4105 with Proxmox 8.1 / debian 12 docker LXC running into the same "GPU Hang" syslog. Reading a similar issue #5799 that suggests some kernel issue, I eventually replace Proxmox with 7.4 and debian 11 LXC, and the issue is resolved. My kernel in LXC is 5.15.102-1-pve
. I think the kernel in Proxmox 8.1 is probably 6.5.11-8-pve
.
Same issue here with proxmox 8.1.10 (kernel 6.5.13-5-pve) and frigate running in LXC
Same issue here with proxmox 8.1.10 (kernel 6.5.13-5-pve) and frigate running in LXC
I am running Proxmox as well and it was working on kernel 6.5.13-3. I just updated to Proxmox 8.2.2 today and it uses kernel 6.8.4-2 and Frigate is unable to detect my GPU:
2024-04-24 16:31:55.145428974 Process detector:ov:
2024-04-24 16:31:55.148000145 Traceback (most recent call last):
2024-04-24 16:31:55.148038397 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-04-24 16:31:55.148044573 self.run()
2024-04-24 16:31:55.148051060 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-04-24 16:31:55.148063664 self._target(*self._args, **self._kwargs)
2024-04-24 16:31:55.148072342 File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2024-04-24 16:31:55.148131608 object_detector = LocalObjectDetector(detector_config=detector_config)
2024-04-24 16:31:55.148163715 File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
2024-04-24 16:31:55.148169848 self.detect_api = create_detector(detector_config)
2024-04-24 16:31:55.148175713 File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-04-24 16:31:55.148180348 return api(detector_config)
2024-04-24 16:31:55.148185865 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 32, in __init__
2024-04-24 16:31:55.148190875 self.interpreter = self.ov_core.compile_model(
2024-04-24 16:31:55.148196953 File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 399, in compile_model
2024-04-24 16:31:55.148205619 super().compile_model(model, device_name, {} if config is None else config),
2024-04-24 16:31:55.148213584 RuntimeError: Failed to create plugin /usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_gpu_plugin.so for device GPU
2024-04-24 16:31:55.148286612 Please, check your environment
2024-04-24 16:31:55.148294444 Check 'error_code == 0' failed at src/plugins/intel_gpu/src/runtime/ocl/ocl_device_detector.cpp:194:
2024-04-24 16:31:55.148303564 [GPU] No supported OCL devices found or unexpected error happened during devices query.
2024-04-24 16:31:55.148309311 [GPU] Please check OpenVINO documentation for GPU drivers setup guide.
2024-04-24 16:31:55.148365420 [GPU] clGetPlatformIDs error code: -1001
Nothing has changed in my config, and it's rather simple for detectors:
detectors:
ov:
type: openvino
device: GPU
I'm guessing it's something to do with how the kernel may be exposing devices, and/or with an update required to some libs in the Frigate container?
FYI - if I change to device: AUTO
, it works fine. I can't tell if it's actually using the GPU or not since vainfo doesn't work for me (no privs), but I'm guessing not... inference is up at around double what it used to be.
@esand I think it is most likely that your issue is not related to frigate. You probably need to reconfigure the GPU passthrough to LXC/VM after Proxmox upgrade.
@esand I think it is most likely that your issue is not related to frigate. You probably need to reconfigure the GPU passthrough to LXC/VM after Proxmox upgrade.
The /dev/dri devices are still visible in both the linux container and the frigate container. Permissions are correct and I've still got hwaccel working just fine. As far as I'm aware, no changes in Proxmox 8.2.2 impacted hardware passthrough configurations and my LXC still boots up just fine (it would error out on a bad config). I also have other devices that I do passthrough with in other containers and those are still functioning fine.
Same observation as @esand. Hwaccel working fine and other cotainers working with the gpu (e.g. plex transcoding).
I just upgraded proxmox to 8.2.2 and have the same issue as described above with intel gpu passthrough
2024-04-26 10:58:11.974474134 [2024-04-26 10:58:11] detector.ov INFO : Starting detection process: 306 2024-04-26 10:58:12.048736394 Process detector:ov: 2024-04-26 10:58:12.049568318 Traceback (most recent call last): 2024-04-26 10:58:12.049570624 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap 2024-04-26 10:58:12.049571729 self.run() 2024-04-26 10:58:12.049572957 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run 2024-04-26 10:58:12.049574044 self._target(*self._args, **self._kwargs) 2024-04-26 10:58:12.049575166 File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector 2024-04-26 10:58:12.049576263 object_detector = LocalObjectDetector(detector_config=detector_config) 2024-04-26 10:58:12.049594351 File "/opt/frigate/frigate/object_detection.py", line 53, in init 2024-04-26 10:58:12.049595625 self.detect_api = create_detector(detector_config) 2024-04-26 10:58:12.049596786 File "/opt/frigate/frigate/detectors/init.py", line 18, in create_detector 2024-04-26 10:58:12.049613578 return api(detector_config) 2024-04-26 10:58:12.049614822 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 32, in init 2024-04-26 10:58:12.049615982 self.interpreter = self.ov_core.compile_model( 2024-04-26 10:58:12.049617125 File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 399, in compile_model 2024-04-26 10:58:12.049628693 super().compile_model(model, device_name, {} if config is None else config), 2024-04-26 10:58:12.049829684 RuntimeError: Failed to create plugin /usr/local/lib/python3.9/dist-packages/openvino/libs/libopenvino_intel_gpu_plugin.so for device GPU 2024-04-26 10:58:12.049831211 Please, check your environment 2024-04-26 10:58:12.049832366 Check 'error_code == 0' failed at src/plugins/intel_gpu/src/runtime/ocl/ocl_device_detector.cpp:194: 2024-04-26 10:58:12.049833459 [GPU] No supported OCL devices found or unexpected error happened during devices query. 2024-04-26 10:58:12.049834541 [GPU] Please check OpenVINO documentation for GPU drivers setup guide. 2024-04-26 10:58:12.049835571 [GPU] clGetPlatformIDs error code: -1001
looks like this is an issue with the kernel included in the new proxmox 8.2.2 https://github.com/blakeblackshear/frigate/discussions/10785
loading the previous kernel in proxmox with the following guide solved my issue until a new proxmox release comes out. https://engineerworkshop.com/blog/how-to-revert-a-proxmox-kernel-update/