frigate
frigate copied to clipboard
blakeblackshear
[Support]: Recordings just stop and won't recover until restart
Describe the problem you are having
I've noticed some weird pattern recently with my Reolink E1 Zoom cameras. I've got the reolink cameras setup as recommended in the documentation but still seeing the behaviour. More often than not, recordings stop around 2:15am and by the time I wake up I can't see any logs so it's difficult to diagnose the issue. I've done some scowering but can't seem to find a way to save logs to a file for review later so I'm wondering if this is achievable so I can check the logs once I've woken up and confirmed it happened again?
Once this has happened, recordings won't resume again until I have restarted the frigate container. Just toggling the recording switch doesn't start it up again.
Version
0.10.0-bfecee9
Frigate config file
logger:
default: debug
logs:
peewee: info
mqtt:
host: <redacted>
port: 1883
topic_prefix: frigate
client_id: frigate
user: <redacted>
password: <redacted>
stats_interval: 60
cameras:
living_room:
ffmpeg:
inputs:
- path: http://<ip>/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pass
roles:
- rtmp
- record
- path: http://<ip>/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pass
roles:
- detect
output_args:
record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
input_args:
- -avoid_negative_ts
- make_zero
- -fflags
- nobuffer+genpts+discardcorrupt
- -flags
- low_delay
- -strict
- experimental
- -analyzeduration
- 1000M
- -probesize
- 1000M
- -rw_timeout
- "5000000"
detect:
width: 896
height: 672
fps: 5
max_disappeared: 25
record:
enabled: true
retain:
days: 7
mode: all
events:
pre_capture: 3
post_capture: 3
objects:
- person
- cat
retain:
default: 365
mode: active_objects
snapshots:
enabled: true
timestamp: true
bounding_box: true
retain:
default: 10
objects:
person: 14
objects:
track:
- person
- cat
motion:
mask:
- 2518,50,2516,124,1733,127,1739,48
front_bedroom:
ffmpeg:
inputs:
- path: http://<ip>/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pass
roles:
- rtmp
- record
- path: http://<ip>/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pass
roles:
- detect
output_args:
record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
input_args:
- -avoid_negative_ts
- make_zero
- -fflags
- nobuffer+genpts+discardcorrupt
- -flags
- low_delay
- -strict
- experimental
- -analyzeduration
- 1000M
- -probesize
- 1000M
- -rw_timeout
- "5000000"
detect:
width: 896
height: 672
fps: 5
max_disappeared: 25
record:
enabled: true
retain:
days: 7
mode: all
events:
pre_capture: 3
post_capture: 3
objects:
- person
- cat
retain:
default: 365
mode: active_objects
snapshots:
enabled: true
timestamp: true
bounding_box: true
retain:
default: 10
objects:
person: 14
objects:
track:
- person
- cat
motion:
mask:
- 2518,50,2516,124,1733,127,1739,48
back_bedroom:
ffmpeg:
inputs:
- path: http://<ip>/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pass
roles:
- rtmp
- record
- path: http://<ip>/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pass
roles:
- detect
output_args:
record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
input_args:
- -avoid_negative_ts
- make_zero
- -fflags
- nobuffer+genpts+discardcorrupt
- -flags
- low_delay
- -strict
- experimental
- -analyzeduration
- 1000M
- -probesize
- 1000M
- -rw_timeout
- "5000000"
detect:
width: 896
height: 672
fps: 5
max_disappeared: 25
record:
enabled: true
retain:
days: 7
mode: all
events:
pre_capture: 3
post_capture: 3
objects:
- person
- cat
retain:
default: 365
mode: active_objects
snapshots:
enabled: true
timestamp: true
bounding_box: true
retain:
default: 10
objects:
person: 14
objects:
track:
- person
- cat
motion:
mask:
- 2518,50,2516,124,1733,127,1739,48
feeder:
ffmpeg:
inputs:
- path: http://<ip>:8081
roles:
- record
- detect
output_args:
record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v libx264 -an
input_args:
- -avoid_negative_ts
- make_zero
- -fflags
- nobuffer
- -flags
- low_delay
- -strict
- experimental
- -fflags
- +genpts+discardcorrupt
- -r
- "3" # <---- adjust depending on your desired frame rate from the mjpeg image
- -use_wallclock_as_timestamps
- "1"
detect:
enabled: false
width: 640
height: 480
fps: 5
max_disappeared: 25
record:
enabled: true
retain:
days: 0
mode: all
events:
pre_capture: 3
post_capture: 3
objects:
- person
- cat
retain:
default: 7
mode: active_objects
rtmp:
enabled: false
snapshots:
enabled: false
timestamp: true
bounding_box: true
retain:
default: 10
objects:
person: 14
doorbell:
ffmpeg:
inputs:
- path: rtsp://user:pass@<ip>:554
roles:
- rtmp
- record
- path: rtsp://user:pass@<ip>:554/cam/realmonitor?channel=1&subtype=1
roles:
- detect
output_args:
record: -f segment -segment_time 60 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
detect:
width: 720
height: 576
fps: 5
max_disappeared: 25
record:
enabled: true
retain:
days: 21
mode: all
events:
pre_capture: 5
post_capture: 5
objects:
- person
- cat
retain:
default: 365
mode: active_objects
snapshots:
enabled: true
timestamp: true
bounding_box: true
retain:
default: 10
objects:
person: 14
motion:
mask:
- 62,365,145,376,230,353,497,361,583,430,720,418,720,165,720,127,720,0,0,0,0,318
zones:
driveway:
coordinates: 720,503,720,576,720,576,0,576,77,417,164,358,208,340,342,364,469,372,603,491
objects:
filters:
person:
mask:
- 606,375,541,377,467,364,329,357,248,351,171,333,100,322,0,296,0,0,720,0,720,354
track:
- person
- cat
detectors:
coral:
type: edgetpu
device: usb
birdseye:
enabled: true
mode: continuous
height: 1080
width: 1920
Relevant log output
Can't access
FFprobe output from your camera
Detect URL
Input #0, flv, from 'http://<ip>/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=user&password=pass':
Metadata:
displayWidth : 896
displayHeight : 672
Duration: 00:00:00.00, start: 0.000000, bitrate: N/A
Stream #0:0: Video: h264 (High), yuv420p(progressive), 896x672, 11 fps, 12 tbr, 1k tbn
Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
Recording URL
Input #0, flv, from 'http://<ip>/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=user&password=pass':
Metadata:
displayWidth : 2560
displayHeight : 1920
Duration: 00:00:00.00, start: 0.000000, bitrate: N/A
Stream #0:0: Video: h264 (High), yuv420p(progressive), 2560x1920, 11 fps, 12 tbr, 1k tbn
Stream #0:1: Audio: aac (LC), 16000 Hz, mono, fltp
Frigate stats
{
"back_bedroom": {
"camera_fps": 5.0,
"capture_pid": 241,
"detection_fps": 0.0,
"pid": 231,
"process_fps": 5.0,
"skipped_fps": 0.0
},
"detection_fps": 0.0,
"detectors": {
"coral": {
"detection_start": 0.0,
"inference_speed": 10.18,
"pid": 217
}
},
"doorbell": {
"camera_fps": 5.1,
"capture_pid": 254,
"detection_fps": 0.0,
"pid": 234,
"process_fps": 5.1,
"skipped_fps": 0.0
},
"feeder": {
"camera_fps": 8.3,
"capture_pid": 244,
"detection_fps": 0.0,
"pid": 233,
"process_fps": 8.3,
"skipped_fps": 0.0
},
"front_bedroom": {
"camera_fps": 5.0,
"capture_pid": 237,
"detection_fps": 0.0,
"pid": 228,
"process_fps": 5.0,
"skipped_fps": 0.0
},
"living_room": {
"camera_fps": 5.1,
"capture_pid": 236,
"detection_fps": 0.0,
"pid": 226,
"process_fps": 5.1,
"skipped_fps": 0.0
},
"service": {
"storage": {
"/dev/shm": {
"free": 263.6,
"mount_type": "tmpfs",
"total": 268.4,
"used": 4.8
},
"/media/frigate/clips": {
"free": 182073.5,
"mount_type": "fuseblk",
"total": 1000186.3,
"used": 818112.8
},
"/media/frigate/recordings": {
"free": 182073.5,
"mount_type": "fuseblk",
"total": 1000186.3,
"used": 818112.8
},
"/tmp/cache": {
"free": 17391.2,
"mount_type": "overlay",
"total": 121029.6,
"used": 97446.3
}
},
"temperatures": {},
"uptime": 21286,
"version": "0.10.0-bfecee9"
}
}
Operating system
Debian
Install method
Docker CLI
Coral version
USB
Network connection
Wired
Camera make and model
Reolink E1 Zoom
Any other information that may be helpful
Relevant cameras in the config file are front_bedroom back_bedroom and living_room
Interesting. I noticed that setting the retain mode to all for recordings solved the problem for you but mine are all already on all for the offending cameras. Also you said that yours occurs when it gets busy whereas my troublesome cameras are indoor cameras that see next to no motion all night, and the detect is turned off almost always.
It's been helpful to see the debugging process of your issue though so I will check out the /tmp/cache directory on my machine to see if the recordings are there and getting cleaned up or not. Thanks!
I'm gonna take a look tomorrow morning at the cache to see whats in there if the crash happens. Recording is working right now so all is fine.
Trying to decypher a bit. I can confirm that it happened again last night on 2/3 cameras, recording stopped for them both within a minute of one another. I can see in the directory /var/lib/docker/containers/<id>/mounts/shm/ the following files, all of which are constantly updating.
I looked at netdata around the time the recordings stopped and I see no spike in CPU or RAM usage. Detect is off for all cameras.
Is there a way to make the logs save to a file so I can see what happens when the recordings stop?
Interesting. I noticed that setting the retain mode to
allfor recordings solved the problem for you but mine are all already onallfor the offending cameras. Also you said that yours occurs when it gets busy whereas my troublesome cameras are indoor cameras that see next to no motion all night, and the detect is turned off almost always.It's been helpful to see the debugging process of your issue though so I will check out the
/tmp/cachedirectory on my machine to see if the recordings are there and getting cleaned up or not. Thanks!
Just a thought... but I see the difference between our configs is that I'm setting the retain mode GLOBALLY, so when I set retain mode to "all", its doing so for all cameras. I wonder if setting the retain mode to 'all' GLOBALLY (rather than for individual cameras) would make a difference? It seems like something strange is going on with the record process.
@joeShuff the files in /dev/shm are the raw decoded frames from the detect stream used for processing. They shouldn't have any impact on recording at all. You should be looking at the files in /tmp/cache instead.
Thanks @blakeblackshear - I have found the location of the /tmp/cache on my host. For future reference/readers I did it by running sudo find / -name "*.tflite" and then checking the resulting locations that had /merged/ toward the end of the path. I did a system reboot earlier so all the recordings are working fine right now and I see that in the files in /tmp/cache. I'll check in when the recording next fails.
Once again it happened overnight to 2/3 cameras (living_room and front_bedroom). The state is as follows. All my cameras still work with live view so frigate is picking up the stream.
In /tmp/cache I can see that the last file for the 2 offending cameras was when recordings stopped.
When I turn recordings on for the offending cameras, I see nothing in the logs other than the messages about recordings being toggled. Also nothing changes in /tmp/cache, the files are still from 2am.
I have seen this issue with Reolinks several times when people were using the RTMP streams. This is the first time I have seen it with the http streams. Basically the camera stops sending data, but doesn't disconnect. This causes ffmpeg to think more data is coming. The rw_timeout parameter in the input args is supposed to fix it, but it's not for some reason.
When recordings are turned off in Frigate, it doesn't stop the ffmpeg process from writing files to /tmp/cache. Frigate just discards them instead of copying them to the recordings folder.
Live view still works because that uses the detect stream which is a separate input with a separate ffmpeg process. It's just the main stream that is hanging.
so it's weird that it only happens on 2/3 of the cameras consistently. I've checked the settings of the cameras and they are all identical and its not even the closest one to the router that works fine.
I tried the rtmp stream from frigate at:
rtmp://${serverAddress.host}:1935/live/${camera.name}
and as expected it doesn't work for the 2 cameras that have failed, but does for the others.
I also checked the direct http streams and they're still up running fine.
sounds like a tricky one...
I think the only way to address this may be to force restart ffmpeg if there havent been any recordings files generated for a while. It's possible that the ffmpeg version in frigate 0.11.0 will end up fixing this.
When the tcp stream is broken (for any reason, camera reboot, router reboot, etc) between ffmpeg and the camera, ffmpeg just continues silently but stops recording anything and doesn't recover. I have to set -stimeout 30000000 (with transport as tcp) for ffmpeg to exit when the camera connection breaks.
I'm looking at trying this because I have similar issue where live preview and records stop but cannot find anything in the logs to say it's the same issue as this.
@mikegleasonjr I'm willing to give this a go however the docs say there is a default -stimeout 5000000 so does it work for you even without setting it albeit at a longer period before it resets?
So
All of my cameras are hardwired. Except my front doorbell camera which is wifi.
So if I unplug a camera, and plug it back in, events still happen. But recordings never recover. Ie camera disconnected but comes back, but recordings 24/7 never recover. Same for the wifi front door bell, occasional disconnect, but recordings never recover. Events and Snapshots continue, but there's no recording for any of the events.
The other scenario - extended power loss pass UPS, frigate will boot fast, the cameras not quite as fast, frigate processes events once the cameras come online, but recordings never happen. Frigate must be restarted.
Please implement something like you mentioned above for frigate to check the last recording time to recover ffmpeg
@Deckoz2302 you need to share your config otherwise it could easily be a misconfiguration
Its exactly as @mikegleasonjr mentioned on July 1st. When there is a TCP event. My recording and config is perfect, and records I keeps 30 days of logs. I discovered this adding a new camera to an existing cat6 line with a poe splitter. Noticed the camera doesn't recover after a TCP event, which lead me to testing full power failure and recovery, (if frigate comes up before the cameras) it captures events and Snapshots from the cameras but recordings never restart.
TCP events are the ONLY time I've lost recordings in 100+ days on hardwired cameras
mqtt:
host: my.domain.com
user: username
password: password
objects:
# Optional: list of objects to track from labelmap.txt (default: shown below)
track:
- person
- dog
- car
- cat
- bicycle
database:
path: /media/frigate.db
record:
enabled: True
retain:
days: 14
mode: all
events:
retain:
default: 10
mode: active_objects
objects:
dog: 2
person: 7
car: 7
pre_capture: 5
post_capture: 15
detectors:
coral:
type: edgetpu
device: usb
coral1:
type: edgetpu
device: pci
coral2:
type: edgetpu
device: pci:1
coral3:
type: edgetpu
device: pci:2
coral4:
type: edgetpu
device: pci:3
ffmpeg:
global_args: -hide_banner -loglevel warning
hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp
output_args:
detect: -f rawvideo -pix_fmt yuv420p
record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
rtmp: -c copy -f flv
rtmp:
enabled: True
environment_vars:
LIBVA_DRIVER_NAME: i965
cameras:
front_doorbell:
ui:
order: 5
ffmpeg:
inputs:
- path: rtsp://username:password@ipaddress:554/cam/realmonitor?channel=1&subtype=0 # <----- Update for your camera
roles:
- record
- path: rtsp://username:password@ipaddress:554/cam/realmonitor?channel=1&subtype=1
roles:
- detect
- rtmp
rtmp:
enabled: true
objects:
track:
- person
detect:
width: 720
height: 576
fps: 10
record:
enabled: true
events:
required_zones:
- front_porch
snapshots:
enabled: true
bounding_box: true
crop: true
timestamp: false
height: 500
retain:
default: 5
required_zones:
- front_porch
mqtt:
timestamp: false
bounding_box: false
crop: true
quality: 100
height: 500
zones:
front_porch:
coordinates: 127,576,720,576,720,92,625,125,227,44,215,384
objects:
- person
motion:
mask:
- 0,384,0,576,72,576,177,392
front_left:
ui:
order: 1
ffmpeg:
inputs:
- path: rtsp://username:password@ipaddress:554/cam/realmonitor?channel=1&subtype=0 # <----- Update for your camera
roles:
- record
- path: rtsp://username:password@ipaddress:554/cam/realmonitor?channel=1&subtype=1 # <----- Update for your camera
roles:
- rtmp
- path: rtsp://username:password@ipaddress:554/cam/realmonitor?channel=1&subtype=2 # <----- Update for your camera
roles:
- detect
rtmp:
enabled: true
detect:
width: 1280
height: 720
fps: 5
record:
enabled: true
events:
required_zones:
- front_walkway
- driveway_entrance
- driveway
- sidewalk
- street
snapshots:
enabled: true
bounding_box: true
crop: true
timestamp: false
height: 500
retain:
default: 5
required_zones:
- front_walkway
- driveway_entrance
- driveway
- sidewalk
- street
mqtt:
timestamp: false
bounding_box: false
crop: true
quality: 100
height: 500
motion:
mask:
- 93,87,272,53,458,35,766,27,974,44,1165,76,1026,0,0,0,0,110
- 1052,106,941,105,1055,153,998,157,1146,246,1280,330,1280,136,1211,119
zones:
front_walkway:
coordinates: 717,720,325,720,275,587,280,526,277,472,259,435,275,387,339,345,426,323,610,292,596,373,499,392,441,421,474,480,559,520
objects:
- person
driveway:
coordinates: 1268,380,1177,536,1230,574,1119,720,995,720,698,486,596,374,606,297,736,249,737,220,684,193,897,166
objects:
- person
- car
- dog
driveway_entrance:
coordinates: 619,157,682,193,896,167,833,136,799,107,524,127
objects:
- car
sidewalk:
coordinates: 233,219,440,181,613,157,746,144,840,135,1015,129,1056,153,806,177,677,192,487,223,245,271,0,335,0,281
objects:
- person
street:
coordinates: 0,213,249,155,501,118,707,101,894,96,1184,114,1105,67,1010,53,870,39,687,34,524,36,329,54,137,87,0,121
objects:
- car
@Deckoz2302 as it shows in your config:
ffmpeg:
global_args: -hide_banner -loglevel warning
hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp
output_args:
detect: -f rawvideo -pix_fmt yuv420p
record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
rtmp: -c copy -f flv
you have overwritten the input_args and have not included the timeout which is why this is happening. Simply add timeout back in and it will solve the issue
@Deckoz2302 as it shows in your config:
ffmpeg: global_args: -hide_banner -loglevel warning hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp output_args: detect: -f rawvideo -pix_fmt yuv420p record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac rtmp: -c copy -f flvyou have overwritten the input_args and have not included the
timeoutwhich is why this is happening. Simply addtimeoutback in and it will solve the issue
Can you share where in the documentation it states this? As the documentation reads as these are additional args, not full replacement of existing args?
As the documentation reads as these are additional args, not full replacement of existing args?
They are replacements, if they were additional then the args would be doubled up since the ones you included are also part of the default list.
an example of this is found here: https://docs.frigate.video/faqs#audio-in-recordings where it is suggested that you have to replace all the args to enable audio (instead of just adding -c:a aac)
And to be clear, in 0.12 we are going to add another check in frigate directly to catch this happening, but it's important to understand that it is most often a config issue due to missing timeout in overridden args and only rarely an actual issue with ffmpeg itself.
I am having similar issue, the recording stop after a day or less until I restart the camera or Frigate container. Running on bare metal with Reolink 510WA , RTMP protocol, using the following camera config:
ffmpeg:
hwaccel_args: -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format yuv420p
input_args: -avoid_negative_ts make_zero -fflags nobuffer -flags low_delay -strict experimental -fflags +genpts+discardcorrupt -rw_timeout 5000000
There are no errors in the docker container logs except "No recordings found for the requested time range" which is due to when looking at recordings in the GUI.
For some reason reolink seems to have this issue, it's already been fixed for 0.12
When is 0.12 targeted to be released? If the fix is already in the dev branch can I build the docker image (sorry a bit off topic now) ?
You can follow the progress here: https://github.com/blakeblackshear/frigate/pull/4055 there will be a beta before release
You can use the dev images from here: https://github.com/blakeblackshear/frigate/pkgs/container/frigate but do be aware that there may be bugs and the changes are undocumented (besides the source code itself)
Fixed in 0.12 and 0.12 is in beta, thus closing
I am running on Frigate 0.12 and using two Reolink camera's. Unfortunately, I do see that after a number of days (it varies, it might be two or five days), the recordings stop. Only a restart of the Docker fixes the issue. I have the following config:
#database:
#path: /config/frigate.db
mqtt:
enabled: False
birdseye:
enabled: False
detectors:
coral:
type: edgetpu
device: usb
objects:
track:
- person
- car
record:
enabled: True
retain:
days: 10
events:
retain:
default: 10
snapshots:
enabled: True
clean_copy: True
timestamp: False
bounding_box: False
crop: False
height: 175
retain:
default: 10
objects:
person: 20
ffmpeg:
hwaccel_args:
- -c:v:1
- h264_v4l2m2m
input_args: -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -flags low_delay -strict experimental -analyzeduration 1000M -probesize 1000M -rw_timeout 5000000
output_args:
record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy
go2rtc:
streams:
Deurbel:
- http://192.168.2.xxx/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password={REDATED}
- ffmpeg:doorbell#audio=opus
Deurbel_sub:
- http://192.168.2.xxx/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password={REDATED}
cameras:
Deurbel:
ffmpeg:
output_args:
record: preset-record-generic-audio-copy
inputs:
- path: rtsp://127.0.0.1:8554/Deurbel?video=copy&audio=aac
input_args: preset-rtsp-restream
roles:
- record
- path: rtsp://127.0.0.1:8554/Deurbel_sub?video=copy&audio=aac
input_args: preset-rtsp-restream
roles:
- detect
detect:
width: 2560
height: 1920
fps: 5
motion:
mask:
- 537,672,0,672,551,100
Carport:
ffmpeg:
inputs:
- path: http://192.168.2.xxx/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password={REDATED}
roles:
- record
- path: http://192.168.2.xxx/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password={REDATED}
roles:
- detect
detect:
width: 2560
height: 1920
fps: 5
Did I overlooked a part or am I still seeing unstable behavior?
