frigate icon indicating copy to clipboard operation
frigate copied to clipboard
Published 3 days ago • verified publisher icon blakeblackshear

[Support]: very slow startup time

Open billyburly opened this issue 1 year ago • 4 comments

Describe the problem you are having

frigate startup is taking a very long time. from a docker-compose up -d to frigate actually running takes about 5 minutes

Version

0.13.0-c35c7da

Frigate config file

objects:
  track:
    - person
    - cat
  filters:
    cat:
      threshold: 0.4
    
cameras:
  front_side_overview:
    ffmpeg:
      inputs:
        - path: XXX
          roles:
            - detect
        - path: XXX
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 1237,25,1237,62,878,62,878,25

  driveway_side_overview:
    ffmpeg:
      inputs:
        - path: XXX
          roles:
            - detect
        - path: XXX
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 1237,25,1237,62,878,62,878,25
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500
  
  server_closet:
    ffmpeg:
      inputs:
        - path: XXX
          roles:
            - detect
        - path: XXX
          roles:
            - record
    detect:
      width: 640
      height: 480
      fps: 5
    motion:
      mask:
        - 640,21,640,0,640,40,454,42,453,21
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500

  front_yard:
    ffmpeg:
      inputs:
        - path: XXX
          roles:
            - detect
        - path: XXX
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 1238,27,1238,62,876,62,876,27 
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500

  front_porch:
    ffmpeg:
      inputs:
        - path: XXX
          roles:
            - detect
        - path: XXX
          roles:
            - record
    motion:
      mask:
        - 1238,27,1238,62,876,62,876,27
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500

  side_door:
    ffmpeg:
      inputs:
        - path: XXX
          roles:
            - detect
        - path: XXX
          roles:
            - record
    motion:
      mask:
        - 1237,25,1237,62,878,62,878,25
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500

  driveway_right:
    ffmpeg:
      inputs:
        - path: XXX
          roles:
            - detect
        - path: XXX
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 1238,27,1238,62,876,62,876,27
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500
      
  driveway_left:
    ffmpeg:
      inputs:
        - path: XXX
          roles:
            - detect
        - path: XXX
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 5
    motion:
      mask:
        - 1238,27,1238,62,876,62,876,27 
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500

  garage:
    ffmpeg:
      inputs:
        - path: XXX
          roles:
            - detect
        - path: XXX
          roles:
            - record
    detect:
      width: 1280
      height: 720
      fps: 5
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500

  garage_back:
    ffmpeg:
      inputs:
        - path: XXX
          roles:
            - detect
        - path: XXX
          roles:
            - record
    detect:
      width: 640
      height: 480
      fps: 5
    motion:
      mask:
        - 640,21,640,0,640,40,454,42,453,21
    mqtt:
      timestamp: False
      bounding_box: False
      crop: True
      quality: 100
      height: 500

      
snapshots:
  enabled: True
  timestamp: True
  bounding_box: True

birdseye:
  enabled: True

record:
  enabled: True
  retain:
    days: 7
    mode: all
  events:
    retain:
      default: 60

mqtt:
  host: XXX
  user: XXX
  password: XXX

detectors:
  coral:
    type: edgetpu
    device: pci

ffmpeg:
  output_args:
    record: preset-record-generic-audio-aac
  hwaccel_args: preset-vaapi

Relevant log output

2024-01-23 21:25:24.269714751  [INFO] Preparing Frigate...
2024-01-23 21:25:24.270450891  [INFO] Starting NGINX...
s6-rc: info: service legacy-services successfully started
2024-01-23 21:25:24.280871881  [INFO] Preparing new go2rtc config...
2024-01-23 21:25:24.337916400  [INFO] Starting Frigate...
2024-01-23 21:25:24.552923070  [INFO] Starting go2rtc...
2024-01-23 21:25:24.630260295  21:25:24.630 INF go2rtc version 1.8.4 linux/amd64
2024-01-23 21:25:24.630525987  21:25:24.630 INF [rtsp] listen addr=:8554
2024-01-23 21:25:24.630614832  21:25:24.630 INF [webrtc] listen addr=:8555
2024-01-23 21:25:24.635623760  21:25:24.635 INF [api] listen addr=:1984
2024-01-23 21:25:24.723173876  2024/01/23 21:25:24 [error] 143#143: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.47.0.230, server: , request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:5002/", host: "nvr:5000"
2024-01-23 21:25:24.723177231  10.47.0.230 - - [23/Jan/2024:21:25:24 -0500] "GET /ws HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"
2024-01-23 21:25:25.444837240  [2024-01-23 21:25:25] frigate.app                    INFO    : Starting Frigate (0.13.0-c35c7da)
2024-01-23 21:25:27.234362445  2024/01/23 21:25:27 [error] 142#142: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.47.0.30, server: , request: "GET /api/stats HTTP/1.1", upstream: "http://127.0.0.1:5001stats", host: "nvr:5000"
2024-01-23 21:25:29.732870200  2024/01/23 21:25:29 [error] 144#144: *5 connect() failed (111: Connection refused) while connecting to upstream, client: 10.47.0.230, server: , request: "GET /ws HTTP/1.1", upstream: "http://127.0.0.1:5002/", host: "nvr:5000"
2024-01-23 21:25:29.732880273  10.47.0.230 - - [23/Jan/2024:21:25:29 -0500] "GET /ws HTTP/1.1" 502 559 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" "-"
2024-01-23 21:25:29.737866611  [2024-01-23 21:25:29] peewee_migrate.logs            INFO    : Starting migrations
2024-01-23 21:25:30.106393242  [2024-01-23 21:25:30] peewee_migrate.logs            INFO    : There is nothing to migrate

...

2024-01-23 21:30:38.377842552  [2024-01-23 21:30:38] frigate.app                    INFO    : Recording process started: 772
2024-01-23 21:30:38.378936862  [2024-01-23 21:30:38] frigate.app                    INFO    : go2rtc process pid: 89
2024-01-23 21:30:38.400320410  [2024-01-23 21:30:38] detector.coral                 INFO    : Starting detection process: 782
2024-01-23 21:30:38.439341004  [2024-01-23 21:30:38] frigate.app                    INFO    : Output process started: 784
2024-01-23 21:30:38.440890225  [2024-01-23 21:30:38] frigate.detectors.plugins.edgetpu_tfl INFO    : Attempting to load TPU as pci
2024-01-23 21:30:38.440920011  [2024-01-23 21:30:38] frigate.detectors.plugins.edgetpu_tfl INFO    : TPU found
2024-01-23 21:30:38.703325532  [2024-01-23 21:30:38] frigate.app                    INFO    : Camera processor started for front_side_overview: 835
2024-01-23 21:30:38.703471694  [2024-01-23 21:30:38] frigate.app                    INFO    : Camera processor started for driveway_side_overview: 836
2024-01-23 21:30:38.703493631  [2024-01-23 21:30:38] frigate.app                    INFO    : Camera processor started for server_closet: 838
2024-01-23 21:30:38.707317561  [2024-01-23 21:30:38] frigate.app                    INFO    : Camera processor started for front_yard: 841

FFprobe output from your camera

N/A

Frigate stats

No response

Operating system

Other Linux

Install method

Docker Compose

Coral version

PCIe

Network connection

Wired

Camera make and model

N/A

Any other information that may be helpful

No response

billyburly avatar Jan 24 '24 02:01 billyburly

Does this happen every time?

NickM-27 avatar Jan 24 '24 02:01 NickM-27

Unfortunately no. I haven't been able to pinpoint a trigger.

billyburly avatar Jan 24 '24 02:01 billyburly

Judging by the logs, either something in init_database or init_recording_manager is slow.

Looking at the former, it will perform vacuuming (https://www.sqlite.org/lang_vacuum.html) on the sqlite DB on startup if it hasn't happened in the last two weeks. This could potentially cause a delay, if you had a large Frigate DB. What's the output of ls -lh and cat .vacuum in your /config directory? Slow network storage could also be a culprit, if you're not running locally.

Otherwise, sharing the output of ps -ef on your host while Frigate is "stuck" starting up would show which processes are already running, which would help pinpoint further e.g. whether the RecordingMaintainer process is in existence yet.

leccelecce avatar Jan 24 '24 21:01 leccelecce

I'm currently managing 3 frigate setups and they all have very similar configuration (proxmox/lxc/coral/docker) with the exception of the amount of cameras and brands on each setup... Each setup crashes due to a memory leak at some point that I have not been able to pinpoint but something that's has become obvious is the time that takes to boot when the database has store many new events and no new requests has been sent to "http://frigate_server_ip:5000/events" apparently this request keeps the sqlite cache current allowing normal boot times, shorter vacuum times and preventing memory leaks.

I'm making reference to my post where I wonder if support to a more robust database is in the roadmap here although seems like that's not an option.

The only way I've been able to keep normal frigate's boot time (due to long vacuum process or sqlite recovery process) is by running a cron job every few minutes to "http://frigate_server_ip:5000/event" . The time between request depends on how often new events are being created . This is not ideal and this should not happen to begin with... but I have not been able to properly debug this 2 problems as the server becomes inaccessible due to high cpu/memory usage or while the server is booting as the frigate container is not fully up and seems like this is not happening to anyone in the frigate dev team.

ronluna avatar Feb 11 '24 14:02 ronluna

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

github-actions[bot] avatar Mar 13 '24 00:03 github-actions[bot]