frigate icon indicating copy to clipboard operation
frigate copied to clipboard

[Detector Support]: Frigate terminates after about a minute with two Corals

Open tbclark3 opened this issue 1 year ago • 5 comments

Describe the problem you are having

I would like to use two Corals--one for Frigate and one for Compreface. With Frigate running I can plug in the second Coral and it makes no difference. However, after restarting Frigate, it comes up, attaches a Coral, runs for about a minute and shuts down. I tried limiting its devices by mapping /dev/bus/usb/004 and the same thing happened. I don't see how it can be a power issue because the Corals are on different hubs, one of which is an add-on pcie card. So I tried setting up Frigate to use both Corals, following the configuration instructions. It comes up, shows all the cameras, shows both TPU detectors and uses both detectors, then shuts down after about a minute.

Version

0.13.1-34FB1C2

Frigate config file

mqtt:
  host: mqtt.ferree-clark.org
  port: 1883
  topic_prefix: frigate
  client_id: frigate
  user: frigate
  password: xxxxxxx

detectors:
  coral1:
    type: edgetpu
    device: usb:0
  coral2:
    type: edgetpu
    device: usb:1

database:
  path: /media/frigate/frigate.db
model:
  path: /edgetpu_model.tflite
  labelmap_path: /labelmap.txt
  width: 320
  height: 320
  labelmap:
    2: vehicle
    3: vehicle
    7: vehicle
    16: animal
    17: animal
    18: animal
    19: animal
    20: animal
    22: animal

logger:
  default: info

birdseye:
  enabled: True
  width: 1280
  height: 720
  quality: 8
  mode: objects
  restream: true

ffmpeg:
  global_args: -hide_banner -loglevel warning
  hwaccel_args: preset-vaapi
  output_args:
    record: preset-record-generic-audio-copy

detect:
  width: 704
  height: 480
  fps: 6
  enabled: True
  stationary:
    interval: 50
    threshold: 50
    max_frames:
      objects:
        person: 3600
        animal: 1800

motion:
  improve_contrast: False

objects:
  track:
    - person
    - bicycle
    - vehicle
    - animal
  filters:
    person:
      min_area: 1000
      min_score: 0.7
    animal:
      min_score: 0.7
record:
  enabled: True
  retain:
    mode: motion
    days: 10
  events:
    pre_capture: 10
    post_capture: 10
    retain:
      default: 30
      mode: active_objects
      objects:
        animal: 14
  expire_interval: 360
snapshots:
  enabled: True
  clean_copy: True
  timestamp: False
  bounding_box: True
  crop: False
  retain:
    default: 30
    objects:
      person: 60
      animal: 14
      vehicle: 60

rtmp:
  enabled: false

live:
  height: 720
  quality: 8

timestamp_style:
  position: "tl"
  format: "%m/%d/%Y %H:%M:%S"
  color:
    red: 255
    green: 255
    blue: 255
  thickness: 2

go2rtc:
  webrtc:
    listen: ":8555"
  streams:
    camera-backyard:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-backyard:554/cam/realmonitor?channel=1&subtype=0"
      - "ffmpeg:camera-backyard#audio=opus"
    camera-backyard_sub:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-backyard:554/cam/realmonitor?channel=1&subtype=1"
      - "ffmpeg:camera-backyard_sub#audio=opus"
    camera-street:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-street:554/cam/realmonitor?channel=1&subtype=0"
      - "ffmpeg:camera-street#audio=opus"
    camera-street_sub:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-street:554/cam/realmonitor?channel=1&subtype=1"
      - "ffmpeg:camera-street_sub#audio=opus"
    camera-backdoor:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-backdoor:554/cam/realmonitor?channel=1&subtype=0"
      - "ffmpeg:camera-backdoor#audio=opus"
    camera-backdoor_sub:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-backdoor:554/cam/realmonitor?channel=1&subtype=1"
      - "ffmpeg:camera-backdoor_sub#audio=opus"
    camera-alley:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-alley:554/cam/realmonitor?channel=1&subtype=0"
      - "ffmpeg:camera-alley#audio=opus"
    camera-alley_sub:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-alley:554/cam/realmonitor?channel=1&subtype=1"
      - "ffmpeg:camera-alley_sub#audio=opus"
    camera-sidewalk:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-sidewalk:554/cam/realmonitor?channel=1&subtype=0"
      - "ffmpeg:camera-sidewalk#audio=opus"
    camera-sidewalk_sub:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-sidewalk:554/cam/realmonitor?channel=1&subtype=1"
      - "ffmpeg:camera-sidewalk_sub#audio=opus"
    camera-frontdoor:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-frontdoor:554/cam/realmonitor?channel=1&subtype=0"
      - "ffmpeg:camera-frontdoor#audio=opus"
    camera-frontdoor_sub:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-frontdoor:554/cam/realmonitor?channel=1&subtype=1"
      - "ffmpeg:camera-frontdoor_sub#audio=opus"
    camera-uppergarage:
      - "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-uppergarage.ferree-clark.org:554/Streaming/Channels/1"
    #camera-uppergarage_sub:
      #- "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-uppergarage.ferree-clark.org:554/Streaming/Channels/2"

cameras:
  camera-backyard:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/camera-backyard
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://localhost:8554/camera-backyard_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
    detect:
      enabled: true
    record:
      enabled: true
    zones:
      backdriveway:
        coordinates: 204,480,402,480,157,320,64,292
      garden:
        coordinates: 192,337,307,419,437,480,571,342,359,172,67,295
      street:
        coordinates: 436,36,570,98,39,284,0,225
    motion:
      mask:
        - 704,480,443,480,704,145
        - 0,480,207,480,0,168
        - 704,158,704,0,388,0
        - 43,76,154,46,292,110,389,161,404,206,258,202,227,256,220,365,116,286,32,247
        - 537,233,544,274,551,304,505,331,481,314,431,311,333,291,354,210,336,154,411,137,469,134
    objects:
      track:
        - person
        - animal
  camera-street:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/camera-street_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://localhost:8554/camera-street
          input_args: preset-rtsp-restream
          roles:
            - record
    detect:
      enabled: true
    objects:
      filters:
        vehicle:
          mask:
            - 46,43,189,0,202,83,47,100
        person:
          max_area: 89999
    record:
      enabled: true
    zones:
      backdriveway:
        coordinates: 395,222,704,480,0,480,0,220,180,124,281,109
      westsidewalk:
        coordinates: 0,162,71,70,127,90,0,217
      garden:
        coordinates: 704,176,485,54,296,86,412,240,704,444
    motion:
      mask:
        - 158,96,222,91,290,84,386,48,396,0,0,0
        - 422,0,676,0,704,199,636,320,539,276,424,206
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 480
      quality: 100
  camera-backdoor:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/camera-backdoor_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://localhost:8554/camera-backdoor
          input_args: preset-rtsp-restream
          roles:
            - record
    detect:
      enabled: true
    objects:
      track:
        - person
        - animal
    record:
      enabled: true
    zones:
      eastsidewalk:
        coordinates: 474,78,625,480,458,480,416,154
    motion:
      mask:
        - 704,418,704,0,515,0
        - 416,480,0,480,0,0,704,0,349,0
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 480
      quality: 100
  camera-alley:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/camera-alley_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://localhost:8554/camera-alley
          input_args: preset-rtsp-restream
          roles:
            - record
    detect:
      enabled: true
    record:
      enabled: true
      events:
        required_zones:
          - frontdriveway
          - alley
    zones:
      frontdriveway:
        coordinates: 704,480,704,408,592,150,115,171,0,226,0,480
        objects:
          - vehicle
          - person
          - animal
      alley:
        coordinates: 658,162,643,116,202,132,99,175
        objects:
          - vehicle
          - person
    motion:
      mask:
        - 219,130,704,110,704,0,186,0,0,0,0,210
    objects:
      filters:
        person:
          mask:
            - 704,127,668,45,489,0,467,73
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 480
      quality: 100
  camera-sidewalk:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/camera-sidewalk_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://localhost:8554/camera-sidewalk
          input_args: preset-rtsp-restream
          roles:
            - record
    detect:
      enabled: true
    objects:
      track:
        - person
        - animal
      filters:
        person:
          max_area: 89999
    record:
      enabled: true
    zones:
      westsidewalk:
        coordinates: 386,140,406,143,272,480,138,480
    motion:
      mask:
        - 0,480,0,0,381,0,370,126,220,325,107,480
        - 415,0,704,0,704,338,704,283,704,480,401,480
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 480
      quality: 100
  camera-frontdoor:
    ffmpeg:
      inputs:
        - path: rtsp://localhost:8554/camera-frontdoor_sub
          input_args: preset-rtsp-restream
          roles:
            - detect
        - path: rtsp://localhost:8554/camera-frontdoor
          input_args: preset-rtsp-restream
          roles:
            - record
    detect:
      enabled: true
    objects:
      track:
        - person
        - animal
        - vehicle
    record:
      enabled: true
    zones:
      frontporch:
        coordinates: 521,241,305,480,148,480,24,366,338,187
      frontsidewalk:
        coordinates: 322,181,479,236,588,76,496,61
      westsidewalk:
        coordinates: 312,480,704,480,704,450,410,371
    motion:
      mask:
        - 0,125,336,41,330,0,0,0
        - 520,365,636,444,704,352,704,219,631,195
        - 465,46,704,51,704,0,465,0
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 480
      quality: 100
  camera-uppergarage:
    ffmpeg:
      inputs:
        #- path:  rtsp://localhost:8554/camera-uppergarage_sub
        #  input_args: preset-rtsp-restream
        #  roles:
        #    - detect
        - path: rtsp://localhost:8554/camera-uppergarage
          input_args: preset-rtsp-restream
          roles:
            - detect
            - record
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
    detect:
      enabled: true
      width: 1280
      height: 720
      fps: 10
    record:
      enabled: true
    objects:
      track:
        - person
        - vehicle
      filters:
        person:
          mask:
            - 1280,441,1280,720,1084,720
    motion:
      mask:
        - 361,64,578,67,578,43,364,46
        - 1280,210,1280,0,701,0
    snapshots:
      enabled: True
    zones:
      garage_doorway:
        coordinates: 0,720,349,720,350,510,0,513
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 720
      quality: 100
  camera-lowergarage:
    ffmpeg:
      inputs:
        - path:  rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-lowergarage.ferree-clark.org:554/play2.sdp
          roles:
            - detect
        - path: rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-lowergarage.ferree-clark.org:554/play1.sdp
          roles:
            - record
    detect:
      enabled: true
      width: 640
      height: 352
      fps: 5
    record:
      enabled: true
    objects:
      track:
        - person
    motion:
      mask:
        - 640,0,640,40,0,42,0,0
    snapshots:
      enabled: False
  camera-basement:
    ffmpeg:
      inputs:
        - path: "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-basement:554/Streaming/Channels/2"
          input_args: preset-rtsp-generic
          roles:
            - detect
        - path: "rtsp://viewer:{FRIGATE_RTSP_PASSWORD}@camera-basement:554/Streaming/Channels/1"
          input_args: preset-rtsp-generic
          roles:
            - record
    detect:
      enabled: true
      width: 640
      height: 480
      fps: 6
    record:
      enabled: true
    objects:
      track:
        - person
      filters:
        person:
          mask:
            - 403,36,533,117,577,0,387,0
    snapshots:
      enabled: False
    birdseye:
      enabled: False
    motion:
      mask:
        - 21,43,267,53,270,0,25,21
  doorbell-front:
    ffmpeg:
      inputs:
        - path: http://ghanms0001:xxxxx@doorbell-front/bha-api/video.cgi
          roles:
            - detect
        - path: rtsp://ghanms0001:xxxxx@doorbell-front:554/mpeg/media.amp
          roles:
            - record
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
      input_args: -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -use_wallclock_as_timestamps 1 -timeout 5000000
    detect:
      enabled: true
      width: 640
      height: 480
      fps: 5
    objects:
      track:
        - person
        - animal
    record:
      enabled: true
    zones:
      eastsidewalk:
        coordinates: 351,441,343,352,281,346,246,439
      frontporch:
        coordinates: 348,445,358,480,0,480,57,394,235,439
    motion:
      mask:
        - 504,480,640,480,640,0,506,0
        - 0,143,84,169,184,150,236,185,345,172,385,0,0,0
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 480
      quality: 100
  doorbell-back:
    ffmpeg:
      inputs:
        - path: http://ghahic0001:xxxxx@doorbell-back/bha-api/video.cgi
          roles:
            - detect
        - path: rtsp://ghahic0001:xxxxx@doorbell-back:554/mpeg/media.amp
          roles:
            - record
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an
      input_args: -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -use_wallclock_as_timestamps 1 -timeout 5000000
    detect:
      enabled: true
      width: 640
      height: 480
      fps: 5
    objects:
      track:
        - person
        - animal
      filters:
        person:
          mask:
            - 0,386,81,389,80,163,0,156
          threshold: 0.65
    record:
      enabled: true
    zones:
      backdriveway:
        coordinates: 521,480,285,315,194,322,0,348,0,480
      westsidewalk1:
        coordinates: 152,261,181,276,132,332,97,333
    motion:
      mask:
        - 640,0,640,70,522,132,341,175,106,201,0,173,0,0
        - 179,263,336,300,334,179,91,204
        - 355,291,428,239,437,164,496,146,513,252,492,290,533,301,528,402,349,342
    mqtt:
      enabled: True
      timestamp: False
      bounding_box: False
      crop: True
      height: 480
      quality: 100

docker-compose file or Docker CLI command

version: "3.9"
services:
  frigate:
    container_name: frigate
    privileged: false
    restart: "unless-stopped"
    image: ghcr.io/blakeblackshear/frigate:stable
    shm_size: "256mb"
    devices:
      - /dev/bus/usb/:/dev/bus/usb/
      - /dev/dri:/dev/dri
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /docker/frigate/config:/config
      - /export/frigate:/media/frigate
      - type: tmpfs 
        target: /tmp/cache
        tmpfs:
          size: 1000000000
    environment:
      FRIGATE_RTSP_PASSWORD: "xxxxx"
      PLUS_API_KEY: "xxxxx"
      LIBVA_DRIVER_NAME: "radeonsi"
    networks:
      br0-network_br0-network:
        ipv4_address:  10.1.10.39
      vlan4-network_vlan4-network:
        ipv4_address: 10.1.4.39
  stunnel:
    container_name: frigate_stunnel
    privileged: false
    restart: "unless-stopped"
    image: dweomer/stunnel
    network_mode: service:frigate
    environment:
      STUNNEL_SERVICE: "frigate"
      STUNNEL_ACCEPT: "443"
      STUNNEL_CONNECT: "5000"
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/letsencrypt/live/ferree-clark.org/fullchain.pem:/etc/stunnel/stunnel.pem:ro
      - /etc/letsencrypt/live/ferree-clark.org/privkey.pem:/etc/stunnel/stunnel.key:ro
networks:
  br0-network_br0-network:
    external: true
  vlan4-network_vlan4-network:
    external: true

Relevant log output

File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 160, in load_delegate
2024-02-11 13:15:03.916344122      delegate = Delegate(library, options)
2024-02-11 13:15:03.916347482    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
2024-02-11 13:15:03.916352772      raise ValueError(capture.message)
2024-02-11 13:15:03.916355092  ValueError
2024-02-11 13:15:03.916388153  
2024-02-11 13:15:03.916391203  During handling of the above exception, another exception occurred:
2024-02-11 13:15:03.916393563  
2024-02-11 13:15:03.916395993  Traceback (most recent call last):
2024-02-11 13:15:03.916398673    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-02-11 13:15:03.916449844      self.run()
2024-02-11 13:15:03.916453104    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-02-11 13:15:03.916456234      self._target(*self._args, **self._kwargs)
2024-02-11 13:15:03.916459224    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
2024-02-11 13:15:03.916510065      object_detector = LocalObjectDetector(detector_config=detector_config)
2024-02-11 13:15:03.916513135    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
2024-02-11 13:15:03.916516195      self.detect_api = create_detector(detector_config)
2024-02-11 13:15:03.916519105  [2024-02-11 13:14:35] frigate.app                    INFO    : Camera processor started for camera-sidewalk: 1346
2024-02-11 13:15:03.916522855    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
2024-02-11 13:15:03.916525265      return api(detector_config)
2024-02-11 13:15:03.916528195    File "/opt/frigate/frigate/detectors/plugins/edgetpu_tfl.py", line 41, in __init__
2024-02-11 13:15:03.916567976      edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config)
2024-02-11 13:15:03.916571536    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
2024-02-11 13:15:03.916574346      raise ValueError('Failed to load delegate from {}\n{}'.format(
2024-02-11 13:15:03.916578216  ValueError: Failed to load delegate from libedgetpu.so.1.0

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Any other information that may be helpful

No response

tbclark3 avatar Feb 11 '24 22:02 tbclark3

it would be good to see the full logs from when frigate starts

NickM-27 avatar Feb 12 '24 12:02 NickM-27

Thanks for looking at it! I am attaching a log file and a screen shot of the two detectors working.

Screenshot from 2024-02-12 10-19-02 frigate.log

tbclark3 avatar Feb 12 '24 18:02 tbclark3

coral1 does not seem to be working, 10 is the default value shown for inference speed until the actual speed is known

NickM-27 avatar Feb 12 '24 18:02 NickM-27

Yet process 1317 has a CPU percentage, and lights on both Corals were flashing during startup. Both Corals are detected according to the logs, and as far as I can tell, the configuration is correct. Is there anything I can do to further investigate?

tbclark3 avatar Feb 12 '24 18:02 tbclark3

Hello, I've got a similar stacktrace since yesterday.

frigate  | 2024-02-14 15:02:04.413213073    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 160, in load_delegate
frigate  | 2024-02-14 15:02:04.413228020      delegate = Delegate(library, options)
frigate  | 2024-02-14 15:02:04.413230632    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
frigate  | 2024-02-14 15:02:04.413232852      raise ValueError(capture.message)
frigate  | 2024-02-14 15:02:04.413235096  ValueError
frigate  | 2024-02-14 15:02:04.413237076
frigate  | 2024-02-14 15:02:04.413239488  During handling of the above exception, another exception occurred:
frigate  | 2024-02-14 15:02:04.413241370
frigate  | 2024-02-14 15:02:04.413243562  Traceback (most recent call last):
frigate  | 2024-02-14 15:02:04.413246155    File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
frigate  | 2024-02-14 15:02:04.413264912      self.run()
frigate  | 2024-02-14 15:02:04.413267748    File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
frigate  | 2024-02-14 15:02:04.413269958      self._target(*self._args, **self._kwargs)
frigate  | 2024-02-14 15:02:04.413272306    File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
frigate  | 2024-02-14 15:02:04.413274685      object_detector = LocalObjectDetector(detector_config=detector_config)
frigate  | 2024-02-14 15:02:04.413277265    File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
frigate  | 2024-02-14 15:02:04.413279585      self.detect_api = create_detector(detector_config)
frigate  | 2024-02-14 15:02:04.413282020    File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
frigate  | 2024-02-14 15:02:04.413284177      return api(detector_config)
frigate  | 2024-02-14 15:02:04.413286737  [2024-02-14 15:01:58] frigate.app                    INFO    : Capture process started for Drive: 382
frigate  | 2024-02-14 15:02:04.413306642    File "/opt/frigate/frigate/detectors/plugins/edgetpu_tfl.py", line 41, in __init__
frigate  | 2024-02-14 15:02:04.413308793      edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config)
frigate  | 2024-02-14 15:02:04.413311109    File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
frigate  | 2024-02-14 15:02:04.413313203      raise ValueError('Failed to load delegate from {}\n{}'.format(
frigate  | 2024-02-14 15:02:04.413315382  ValueError: Failed to load delegate from libedgetpu.so.1.0

Context :

1 USB coral that has been working flawlessly for more than one year. I updated frigate docker container (I run with HA but not as an addon) to latest stable version. It worked without problems for a few days, then the web interface nginx said 502 and this stacktrace runs in loops in the container logs.

I updated the host kernel (Debian 11) recently, and had a power outage, but everything restarted smoothly. I'm looking for clues, I'll reboot and look for kernel messages about the coral to see if it's correctly detected and post them here just for the sake of finding the root cause.

If it doesn't seem to be a hardware problem, I'll post a new issue with my full config etc, and link to this issue because the crash seems the same or related.

Karmak23 avatar Feb 14 '24 14:02 Karmak23

To eliminate any concern for inadequate power, I bought a StarTech.com 2-Port 10Gbps USB-A & USB-C PCIe Card - USB 3.1. If I plug the second Coral into it and restart Frigate, Frigate fails to load the only TPU it can see via devices:/dev/bus/usb/006. Logs are quoted below. Frigate is using device 6-2. The second Coral is device 2-1. Unplugging it and restarting Frigate produces a normal startup. I don't see how this could be internal to Frigate because Frigate shouldn't have any way of knowing about the second Coral. It isn't power. Could the operating system (happens on kernels 6.6.14, 6.7.4 and 6.7.5) be causing this?

Feb 23 10:18:59 clark kernel: usb 2-1: new SuperSpeed USB device number 2 using xhci_hcd Feb 23 10:18:59 clark kernel: usb 2-1: New USB device found, idVendor=1a6e, idProduct=089a, bcdDevice= 1.00 Feb 23 10:18:59 clark kernel: usb 2-1: New USB device strings: Mfr=0, Product=0, SerialNumber=0 Feb 23 10:20:05 clark 9e70ecd2626d[3083]: 2024-02-23 10:20:05.232477576 [2024-02-23 10:19:39] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as usb Feb 23 10:20:59 clark 87494c3dd67d[3083]: 2024-02-23 10:20:59.174416544 [2024-02-23 10:20:33] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as usb Feb 23 10:21:39 clark 87494c3dd67d[3083]: 2024-02-23 10:21:39.284689694 [2024-02-23 10:21:13] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as usb Feb 23 10:21:51 clark kernel: usb 2-1: USB disconnect, device number 2 Feb 23 10:22:08 clark kernel: usb 6-1: reset SuperSpeed USB device number 2 using xhci_hcd Feb 23 10:22:08 clark kernel: usb 6-1: LPM exit latency is zeroed, disabling LPM. Feb 23 10:22:09 clark cc85a491932b[3083]: 2024-02-23 10:22:09.164053239 [2024-02-23 10:22:06] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as usb ...and the cycle repeats

tbclark3 avatar Feb 23 '24 18:02 tbclark3

After some experimentation, I have some explanations.

First, my assumption that Frigate would only know about USB devices mounted with the "devices:" option was incorrect. Docker makes the entire /sys/bus/usb tree available to every container (read-only) even if there is no USB device mapped into the container, so Frigate can enumerate (and get the ID and vendor) all of the USB devices--even those that are not mapped. It can only connect to devices that are mounted in the container.

Second, it appears to me (hopefully not another mistaken assumption) that edgetpu.py enumerates the bus, then tries to connect to the first device it finds with "ID 1a6e:089a Global Unichip Corp." which is what the Coral says when it is first plugged into the USB port. If it doesn't find one, it tries to connect to "ID 18d1:9302 Google Inc." which is what the Coral says after edgetpu.py has loaded a model. Thus, it is necessary to mount the entire /dev/bus/usb system when working with more than one Coral. If only one bus is mounted, eg /dev/bus/usb/001, and there is a second Coral plugged into a different bus, Frigate will fail because it tries first to connect to the device that hasn't been used yet and it can't.

Finally, it is possible to connect to a single Coral from within Frigate even when two are plugged in by specifying only one detector, device usb:0 (or usb:1). It is necessary to make all buses that host Corals available to the container. If all Corals are on bus 1, then /dev/bus/usb/001 works. If they are on different buses, then it has to be /dev/bus/usb or both buses. Connecting to device: usb also works, but I'm guessing it's going to be better to pick one if I want to consistently use the other one for another purpose.

Thus it should be possible to use the second Coral for another purpose such as Compreface.

tbclark3 avatar Feb 28 '24 23:02 tbclark3

For others that discover this post this is how I translated the above to use two USB corals on the same USB bus.

Find the USB bus they are on. lsusb Bus 001 Device 007: ID 18d1:9302 Google Inc. Bus 001 Device 008: ID 18d1:9302 Google Inc.

frigate config: detectors: coral1: type: edgetpu device: usb:0 coral2: type: edgetpu device: usb:1

Docker compose: - /dev/bus/usb/001:/dev/bus/usb/001

nconder avatar Mar 12 '24 02:03 nconder