frigate icon indicating copy to clipboard operation
frigate copied to clipboard

[Support]: 1 out of 4 Edge TPUs not detected

Open deece opened this issue 2 years ago • 5 comments

Describe the problem you are having

I have 2 Dual Edge TPUs installed in MBS's PCIE adapters. I'm running Frigate as a Hassio addon for Home Assistant.

Frigate consistently reports for pci:2 "No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors."

This occurs at pci:2 even when the 2 dual TPUs & adapters are swapped, so I'm not convinced it's a hardware issue.

Linux shows all 4 apex devices:

 dmesg | grep apex
[   11.051976] apex 0000:04:00.0: Apex performance not throttled due to temperature
[   11.051984] apex 0000:03:00.0: Apex performance not throttled due to temperature
[   11.306691] apex 0000:08:00.0: Apex performance not throttled due to temperature
[   11.310702] apex 0000:09:00.0: Apex performance not throttled due to temperature
 ls /sys/class/apex/
apex_0  apex_1  apex_2  apex_3 

I suspect it might be a permissions issue, but the permissions on the devices are all identical:

 ls -l /dev/apex*
crw-rw---- 1 root apex 120, 0 Jul 26 16:17 /dev/apex_0
crw-rw---- 1 root apex 120, 1 Jul 26 16:17 /dev/apex_1
crw-rw---- 1 root apex 120, 2 Jul 26 16:17 /dev/apex_2
crw-rw---- 1 root apex 120, 3 Jul 26 16:17 /dev/apex_3

Version

0.10.1

Frigate config file

detectors:
  coral1:
    type: edgetpu
    device: pci:0
  coral2:
    type: edgetpu
    device: pci:1
  coral3:
    type: edgetpu
    device: pci:2
  coral4:
    type: edgetpu
    device: pci:3
mqtt:
  # Required: host name
  host: homeautomation.lan
  # Optional: port (default: shown below)
  port: 1883
  # Optional: topic prefix (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  topic_prefix: frigate
  # Optional: client id (default: shown below)
  # WARNING: must be unique if you are running multiple instances
  client_id: frigate
  # Optional: user
  user: frigate
  # Optional: password
  # NOTE: Environment variables that begin with 'FRIGATE_' may be referenced in {}.
  #       eg. password: '{FRIGATE_MQTT_PASSWORD}'
  password: xxxxx
  # Optional: interval in seconds for publishing stats (default: shown below)
  stats_interval: 60

record:
  enabled: True
  retain:
    days: 14
  events:
    retain:
      default: 20

cameras:
  # Name of your camera
  rear_hall:
    ffmpeg:
      hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
      output_args:
        detect: -vf transpose=1 -f rawvideo -pix_fmt yuv420p
      inputs:
        - path: rtsp://10.0.2.12:554/user=admin_password=xxxxx_channel=1_stream=0.sdp?real_stream
          roles:
            - detect
            - rtmp
            - record
    motion:
      mask:
        - 1200,452,1201,636,1164,637,1157,449
    zones:
      stairs:
        coordinates: 0,124,168,312,491,304,508,424,463,590,0,720
      hall:
        coordinates: 1280,720,1280,271,834,295,499,300,516,462,472,720
    objects:
      track:
        - person
  garage_west:
    ffmpeg:
      hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
      inputs:
        - path: rtsp://10.0.2.10/user=admin_password=xxxxx_channel=1_stream=0.sdp?real_stream
          roles:
            - detect
            - rtmp
            - record
    zones:
       parking:
         coordinates: 0,223,291,153,748,84,1049,53,1028,21,727,34,542,58,216,110,0,157
       footpath:
         coordinates: 459,124,1005,62,1056,76,1280,76,1280,148,973,217,479,339,0,464,0,224
       driveway:
         coordinates: 1280,720,1280,295,977,225,0,471,0,720
    objects:
      track:
        - person
        - car
        - bicycle
        - boat
        - cat
        - dog
        - backpack
        - handbag
        - suitcase
        - bottle
        - cell phone
  garage_east:
    ffmpeg:
      hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
      inputs:
        - path: rtsp://10.0.2.11/user=admin_password=xxxxx_channel=1_stream=0.sdp?real_stream
          roles:
            - detect
            - rtmp
            - record
    zones:
      parking:
        coordinates: 0,223,291,153,748,84,1049,53,1028,21,727,34,542,58,216,110,0,157
      driveway:
        coordinates: 0,226,0,720,1280,720,1280,307,861,191,669,229,554,232,443,225,366,227,194,174
    objects:
      track:
        - person
        - car
        - bicycle
        - boat
        - cat
        - dog
        - backpack
        - handbag
        - suitcase
        - bottle
        - cell phone
    motion:
      mask:
        - 1280,0,1280,77,1054,74,1039,23,982,33,939,22,768,28,604,48,400,80,204,115,0,159,0,0
  study:
    ffmpeg:
      hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
      inputs:
        - path: rtsp://10.0.2.14:554/user=admin_password=xxxxxx_channel=1_stream=0.sdp?real_stream
          roles:
            - detect
            - rtmp
            - record
    zones:
      study_all:
        coordinates: 0,0,1280,0,1280,720,0,720
    objects:
      track:
        - person
        - backpack
        - handbag
        - suitcase
        - bottle
        - cell phone
    motion:
      mask:
        - 966,73,1280,71,1280,42,957,44
  workshop_east_cam:
    ffmpeg:
      hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
      inputs:
        - path: rtsp://10.0.2.15:554/user=admin_password=xxxxxx_channel=1_stream=0.sdp?real_stream
          roles:
            - detect
            - rtmp
            - record
    zones:
      workshop_west:
        coordinates: 0,315,1280,304,1280,0,0,0
      workshop_east:
        coordinates: 0,331,1172,285,1174,274,1280,158,1280,720,0,720
    objects:
      track:
        - person
        - backpack
        - handbag
        - suitcase
        - bottle
        - cell phone
    motion:
      mask:
        - 808,70,1126,67,1127,38,807,36

Relevant log output

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-07-26 16:18:59] frigate.app                    INFO    : Starting Frigate (0.10.1-83481af)
[2022-07-26 16:18:59] frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
[2022-07-26 16:18:59] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-07-26 16:19:00] peewee_migrate                 INFO    : There is nothing to migrate
[2022-07-26 16:19:00] detector.coral1                INFO    : Starting detection process: 217
[2022-07-26 16:19:00] frigate.edgetpu                INFO    : Attempting to load TPU as pci:0
[2022-07-26 16:19:00] detector.coral2                INFO    : Starting detection process: 218
[2022-07-26 16:19:00] frigate.edgetpu                INFO    : Attempting to load TPU as pci:1
Process detector:coral3:
[2022-07-26 16:19:00] frigate.edgetpu                INFO    : TPU found
[2022-07-26 16:19:00] detector.coral4                INFO    : Starting detection process: 224
[2022-07-26 16:19:00] frigate.edgetpu                INFO    : Attempting to load TPU as pci:3
[2022-07-26 16:19:00] frigate.edgetpu                INFO    : TPU found
[2022-07-26 16:19:00] frigate.app                    INFO    : Output process started: 226
[2022-07-26 16:19:00] frigate.app                    INFO    : Camera processor started for rear_hall: 243
[2022-07-26 16:19:00] frigate.app                    INFO    : Camera processor started for garage_west: 245
[2022-07-26 16:19:00] frigate.app                    INFO    : Camera processor started for garage_east: 246
[2022-07-26 16:19:00] frigate.app                    INFO    : Camera processor started for study: 248
[2022-07-26 16:19:00] frigate.app                    INFO    : Camera processor started for workshop_east_cam: 259
[2022-07-26 16:19:00] frigate.app                    INFO    : Capture process started for rear_hall: 260
[2022-07-26 16:19:00] frigate.app                    INFO    : Capture process started for garage_west: 261
[2022-07-26 16:19:00] frigate.app                    INFO    : Capture process started for garage_east: 262
[2022-07-26 16:19:00] frigate.app                    INFO    : Capture process started for study: 264
[2022-07-26 16:19:00] frigate.app                    INFO    : Capture process started for workshop_east_cam: 265
[2022-07-26 16:19:00] ws4py                          INFO    : Using epoll
[2022-07-26 16:19:00] detector.coral3                INFO    : Starting detection process: 222
[2022-07-26 16:19:00] frigate.edgetpu                INFO    : Attempting to load TPU as pci:2
[2022-07-26 16:19:00] frigate.edgetpu                ERROR   : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.
[2022-07-26 16:19:00] frigate.edgetpu                INFO    : TPU found
[2022-07-26 16:19:00] ws4py                          INFO    : Using epoll
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 160, in load_delegate
    delegate = Delegate(library, options)
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
    raise ValueError(capture.message)
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/edgetpu.py", line 136, in run_detector
    object_detector = LocalObjectDetector(
  File "/opt/frigate/frigate/edgetpu.py", line 44, in __init__
    edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config)
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
    raise ValueError('Failed to load delegate from {}\n{}'.format(
ValueError: Failed to load delegate from libedgetpu.so.1.0
[2022-07-26 16:19:20] frigate.watchdog               INFO    : Detection appears to have stopped. Exiting frigate...
[2022-07-26 16:19:20] frigate.app                    INFO    : Stopping...
[2022-07-26 16:19:20] ws4py                          INFO    : Closing all websockets with [1001] 'Server is shutting down'
[2022-07-26 16:19:20] frigate.events                 INFO    : Exiting event cleanup...
[2022-07-26 16:19:20] frigate.record                 INFO    : Exiting recording cleanup...
[2022-07-26 16:19:20] frigate.stats                  INFO    : Exiting watchdog...
[2022-07-26 16:19:20] frigate.watchdog               INFO    : Exiting watchdog...
[2022-07-26 16:19:20] frigate.object_processing      INFO    : Exiting object processor...
[2022-07-26 16:19:38] frigate.events                 INFO    : Exiting event processor...
[2022-07-26 16:19:38] frigate.record                 INFO    : Exiting recording maintenance...
[2022-07-26 16:19:38] peewee.sqliteq                 INFO    : writer received shutdown request, exiting.
[2022-07-26 16:19:38] root                           INFO    : Waiting for detection process to exit gracefully...
[2022-07-26 16:19:42] root                           INFO    : Waiting for detection process to exit gracefully...
[2022-07-26 16:19:42] root                           INFO    : Waiting for detection process to exit gracefully...
[2022-07-26 16:19:42] root                           INFO    : Waiting for detection process to exit gracefully...

FFprobe output from your camera

n/a

Frigate stats

n/a

Operating system

Other Linux

Install method

HassOS Addon

Coral version

M.2

Network connection

Wired

Camera make and model

various

Any other information that may be helpful

No response

deece avatar Jul 26 '22 07:07 deece

You need to use the beta addon to support 4 corals, currently stable only supports 3

NickM-27 avatar Jul 26 '22 11:07 NickM-27

That might not matter for the full access addon. Is it always the same one that fails? Does it work if you only configure the 3 that work?

blakeblackshear avatar Jul 26 '22 11:07 blakeblackshear

Yes, it works if I comment out the Coral3 block (the one that fails)

I'll try tomorrow commenting out the coral4 block and seeing if it still works in that configuration.

deece avatar Jul 26 '22 11:07 deece

Commenting out Coral4 but leaving Coral3 fails:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[2022-07-27 12:02:40] frigate.app                    INFO    : Starting Frigate (0.10.1-83481af)
[2022-07-27 12:02:40] frigate.app                    INFO    : Creating directory: /tmp/cache
Starting migrations
[2022-07-27 12:02:40] peewee_migrate                 INFO    : Starting migrations
There is nothing to migrate
[2022-07-27 12:02:40] peewee_migrate                 INFO    : There is nothing to migrate
[2022-07-27 12:02:41] detector.coral1                INFO    : Starting detection process: 216
[2022-07-27 12:02:41] frigate.edgetpu                INFO    : Attempting to load TPU as pci:0
[2022-07-27 12:02:41] detector.coral2                INFO    : Starting detection process: 217
[2022-07-27 12:02:41] frigate.edgetpu                INFO    : Attempting to load TPU as pci:1
[2022-07-27 12:02:41] detector.coral3                INFO    : Starting detection process: 221
[2022-07-27 12:02:41] frigate.edgetpu                INFO    : Attempting to load TPU as pci:2
[2022-07-27 12:02:41] frigate.edgetpu                ERROR   : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.
Process detector:coral3:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 160, in load_delegate
    delegate = Delegate(library, options)
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
    raise ValueError(capture.message)
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/frigate/frigate/edgetpu.py", line 136, in run_detector
    object_detector = LocalObjectDetector(
  File "/opt/frigate/frigate/edgetpu.py", line 44, in __init__
    edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config)
  File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
    raise ValueError('Failed to load delegate from {}\n{}'.format(
ValueError: Failed to load delegate from libedgetpu.so.1.0
[2022-07-27 12:02:41] frigate.app                    INFO    : Output process started: 223
[2022-07-27 12:02:41] ws4py                          INFO    : Using epoll
[2022-07-27 12:02:41] frigate.app                    INFO    : Camera processor started for rear_hall: 228
[2022-07-27 12:02:41] frigate.app                    INFO    : Camera processor started for garage_west: 231
[2022-07-27 12:02:41] frigate.app                    INFO    : Camera processor started for garage_east: 233
[2022-07-27 12:02:41] frigate.app                    INFO    : Camera processor started for study: 234
[2022-07-27 12:02:41] frigate.app                    INFO    : Camera processor started for workshop_east_cam: 236

deece avatar Jul 27 '22 02:07 deece

Also interesting that it's pci:2 which I imagine maps to apex_2, figure it would be apex_3 that wasn't found if it was related to https://github.com/blakeblackshear/frigate-hass-addons/pull/72

NickM-27 avatar Jul 27 '22 02:07 NickM-27

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Aug 31 '22 08:08 stale[bot]