blakeblackshear
Adaptive bandwidth playback
Describe what you are trying to accomplish and why in non technical terms With both live and recording players it would be great if the quality could adjust to the available bandwidth offered by the device/connection. Similar to how Netflix or similar streaming services operate.
I do not have any storage limitations so I prefer to maximize the camera bitrate and resolution but this results in slow playback when I'm away from home and using Frigate over WireGuard VPN (recordings playback suffers from this mostly vs live feeds but both to an extent).
Essentially I want to prioritize speed of playback and elimination of delays over quality in the event that the quality of original footage exceeds available bandwidth.
Describe the solution you'd like Automatic detection of available bandwidth and on the fly adjustment of quality based on this.
just adding some thoughts. for adaptive playback of the live stream, the post below describes combining the main and sub streams from a camera to create the different resolutions on demand:
https://www.reddit.com/r/selfhosted/comments/z7likc/nvr_using_rtspipcam_to_hls_with_adaptive_bandwidth/
even for recordings, in my case I would actually prefer this same approach if I could save both streams from a camera to avoid hardware transcoding and to permit adaptive playback. the expense is obviously redundant recordings being stored. however they are low res so its not like the storage amount increase would be anywhere near double. likely a very tolerable amount for a lot of users.
the alternative to this would be on the fly transcoding which i assume wouldn't be guaranteed to work quite as well? but saves on the storage needed.
anyways. no point to this other than sharing what I learned.
just ran across https://github.com/blakeblackshear/frigate/issues/16416 and am confused on why it was closed?
seems like this should be possible either in real time with sufficient hardware (just like plex/jellyfin) or by saving/offering up the substream as the low res version as described above.
Curious @hawkeye217 thoughts
Right, sufficient hardware will be required to do this in real-time. It might be a huge disappointment for many users on low powered SBCs.
the issue was closed as a duplicate. Realistically, at least in my opinion it does not make sense to support both realtime transcoding and combined main + sub stream storage. Both have their advantages and disadvantages but the use cases greatly overlap. I personally don't know what I favor but assuming the other maintainers agree with the premise, one such solution will be implemented at some point. In my opinion storage is cheaper than GPUs
Respectfully, I believe adaptive streaming is a feature that should be supported.
My server's WAN upload speed is 35mbps and loading the recorded streams (most around 5000bps) is painfully slow. Viewing substreams for alerts is not a good option.
I have used on-the-fly transcoding on Blue Iris, Scrypted, and XProtect all on an n95 board and they all support it well. I think many (most?) would choose adaptive on the fly transcoding over storing a third stream.
Thank you for your consideration.
Viewing substreams for alerts is not a good option.
Why not? It is effectively the same thing as transcoding
I think many (most?) would choose adaptive on the fly transcoding over storing a third stream.
we have received a lot of feedback in both directions. Some users want to avoid additional disk usage while others would prefer not putting additional load on their GPU that is already doing a lot of things in some cases.
Why not? It is effectively the same thing as transcoding
It obviously depends on the relative resolution, bitrate, iframe settings, etc., but using the substream is not the same because I want to record at very high quality (since this is a security system) and need to review that media at the highest quality possible for both client and server at any given time.
This problem becomes more significant with 4K cameras that have high recording quality but lower detect quality.
Semantically, I would think the "detect" substream should not be used for "reviewing" security incidents. When reviewing an alert, you should be able to view it at the highest resolution the server and client can support. Using the substream when the network can't support the main stream seems like the wrong solution here.
I believe there's a very good reason adaptive streaming is standardized across all other NVRs.
Has there been any experimentation benchmarking the actual impact of adaptive streaming? There's certainly a CPU/GPU impact, but Frigate could detect "high stress" on the system and fall back to the substream when on-the-fly adaptive transcoding isn't viable for that system.
but using the substream is not the same because I want to record at very high quality (since this is a security system) and need to review that media at the highest quality possible for both client and server at any given time.
Sure... but that is why main and sub stream would be recorded at the same time per this suggestion. At which point, the only way your suggestion is better than just showing the sub stream is if you think the available bandwidth would lead to a perfect middleground between the high res and substream streams.
When reviewing an alert, you should be able to view it at the highest resolution the server and client can support. Using the substream when the network can't support the main stream seems like the wrong solution here.
this distinction seems arbitrary, using a sub stream or an on-the-fly reduced quality stream is effectively the same thing, unless the camera can not provide a usable sub stream in which case the camera is not following Frigate's recommendation anyway. The recommended cameras provide a 1920x1080 and 1280x720 sub streams which should effectively supply this need.
on the fly is more flexible allowing users to have this feature with cameras that don't provide good sub streams.
but Frigate could detect "high stress" on the system and fall back to the substream when on-the-fly adaptive transcoding isn't viable for that system.
like I said, I personally think implementing both makes no sense, this leads to drastically more code to maintain that effectively address the same use case.
lots of cameras also have 3 stream outputs meaning you could literally do a high medium and low all with the same basic code.
overall i think this approach would be easier to implement and more universal since as Nick said above, in this day and age storage is cheap.
if you are storing 4k footage from a camera, even if you elected to store both a 1080p/720p and 360p stream in addition to that its probably only a ~25% increase in storage use at most. as a trade off for this feature i would think 90% of users would be fine with that.
@NickM-27
the only way your suggestion is better than just showing the sub stream is if you think the available bandwidth would lead to a perfect middleground between the high res and substream streams (emphasis mine)
That is absolutely correct and is my intension. That is a good way to distill it.
this distinction seems arbitrary, using a sub stream or an on-the-fly reduced quality stream is effectively the same thing, unless the camera can not provide a usable sub stream in which case the camera is not following Frigate's recommendation anyway. The recommended cameras provide a 1920x1080 and 1280x720 sub streams which should effectively supply this need.
I have looked at the docs quite a bit (very possible I'm missing something), but the Choosing a detect resolution docs suggest a substream should be intended and tuned for detection and perhaps be set to the lowest resolution that is viable for object detection?
Are you recommending that substreams be set as high resolution as possible?
Using the substream for detection and low-res remote viewing may have conflating desires e.g. 5fps may be recommended for detection, but a remote viewer may want 15fps.
@kirsch33
if you are storing 4k footage from a camera, even if you elected to store both a 1080p/720p and 360p stream in addition to that its probably only a ~25% increase in storage use at most. as a trade off for this feature i would think 90% of users would be fine with that.
Yes, if https://github.com/blakeblackshear/frigate/issues/16416 was implemented and I could write a 4k, 1080p, 720p, 480p, 320p stream and that scaled out well, I would do it. Ideally then, the UI could choose the best one out of those streams automatically.
I have looked at the docs quite a bit (very possible I'm missing something), but the Choosing a detect resolution docs suggest a substream should be intended and tuned for detection and perhaps be set to the lowest resolution that is viable for object detection?
I don't know why you are conflating object detection and recording a sub stream, these are two entirely unrelated things. For example with my dahua cameras I use one sub stream as 1280x720 for detection and the second sub stream as 1920x1080 for the live view along with the main stream.
I don't know why you are conflating object detection and recording a sub stream
I may be missing something pretty basic here though. 🤷🏻♂️
Honestly, I would probably lean towards real-time transcoding myself. While I certainly agree that storage is cheap and storing multiple streams simultaneously isn't a terrible solution, I think real-time transcoding will provide a better user experience in more situations (it would better support a perfect middle ground AND it would support those cases where the user has a camera that doesn't support sufficient multiple streams).
And as for stressing the hardware, it's unlikely you'd need to transcode more than one stream at a time, and my 15-year-old Atom based system without a discrete GPU at all, was still able to do that. It seems very unlikely that transcoding would contribute noticeably to the load of an even remotely modern system, even ones without dedicated GPUs (a low-end Intel Core i3 can manage multiple 4k streams).
Then again, if you're looking to support the full frigate experience on integrated systems like a Raspberry Pi, then I suppose I see your point. The low end Broadcom chip they use on the Pi 5 only clocks in at around 1400 passmark, well short of the recommended 12k passmark for transcoding 4k without any hardware acceleration (no QuickSync or VCN like Intel or AMD). Even so, I certainly appreciate that frigate started its life with those low powered solutions, but to limit its feature set based on those devices would, I believe, be a mistake. But that's just my opinion.
@kirk56k well said.
I'm wondering if a Just in Time ABR Proxy could work? Something like https://mistserver.org ?
Or https://github.com/shaka-project/shaka-packager ?
@kirk56k well said.
I'm wondering if a Just in Time ABR Proxy could work? Something like https://mistserver.org ?
Or https://github.com/shaka-project/shaka-packager ?
An interface for swapping out stream urls in the UI could be an interesting compromise. Then you could configure an adaptive proxy and specify the proxy url for each camera that gets rendered into the UI (of course you'd need to manage your proxy config to pull the streams from Frigate and re-encode)
Same concept as supporting a CDN at a different URL in a web app
That also gives the flexibility to run the re-encode on different hardware
+1 for real time transcoding, absolutely. Due to limitations on my hardware my sub stream resolution and bitrate are greater than my upload speeds can stream without significant lagging. My old i3 frigate server however transcodes like a champ. I appreciate the value of facilitating a solution for the widest possible user base over transcoding, but there are those of us who'll not be able to use the feature due to hardware limitations on the camera side. A single server that can transcode is much cheaper than replacing a bunch of cameras.
Why not both multiple stream recording and playback transcoding, with settings to enable them per camera.
In my case, I have a high res H.265 main stream on some cameras, and need one of these solutions to view recordings on my tablet that doesn't support H.265.
Why not both multiple stream recording and playback transcoding, with settings to enable them per camera.
higher complexity in implementation and maintaining the code, more confusion for a user over which should be chosen, more complication for the webUI to know which option to choose when both are available.
We haven't began looking into this in detail yet so there could still be pitfalls that make one a much more sane choice as well.
In my case, I have a high res H.265 main stream on some cameras, and need one of these solutions to view recordings on my tablet that doesn't support H.265.
that would be covered by either solution
Why not both multiple stream recording and playback transcoding, with settings to enable them per camera.
higher complexity in implementation and maintaining the code, more confusion for a user over which should be chosen, more complication for the webUI to know which option to choose when both are available.
We haven't began looking into this in detail yet so there could still be pitfalls that make one a much more sane choice as well.
In my case, I have a high res H.265 main stream on some cameras, and need one of these solutions to view recordings on my tablet that doesn't support H.265.
that would be covered by either solution
Once the ability to record / select multiple streams or transcode on playback, the enable/disable is trivial in comparison.
I'd be fine with a "mode" setting so that only one option per camera can be selected.
Personally I don't see the effort of implementing both options being worth it, but again we haven't extensively looked into this, so can't say what the exact implementation will look like
In my case, the camera has a H.265 main stream that can't be played on my tablet. It also has a H.264 sub stream that's being used for detect. The compute resources needed to store the second stream and select on playback are much lower in comparison to transcode on the fly.
If I only had the H.265 stream or short on storage, then I would need to use the transcode on the fly.
I think both options are needed, with priority of implementation being the question.
The compute resources needed to store the second stream and select on playback are much lower in comparison to transcode on the fly.
like many users have pointed out, transcoding these days can be done without many resources on even 6-8 year old platforms with an integrated GPU, so in most cases downscaling and transcoding should not be a large burden. It offers the most flexibility in terms of getting a stream that fits the desired needs. This is also similar to how Jellyfin works so we will likely be able to follow a similar approach if transcoding is the desired way to go.
For the h264 issue can't you already just add something like the following to your primary stream in the go2rtc section to have it auto transcode to h264 hardware accelerated or define them as separate streams one that is H264 compatible?
go2rtc:
streams:
my-high-res:
- rtsp://creds:creds@hostname/h265Preview_01_main
- ffmpeg:my-high-res#video=h264#audio=copy#hardware
or
go2rtc:
streams:
my-high-res-h265:
- rtsp://creds:creds@hostname/h265Preview_01_main
my-high-res-h264:
- ffmpeg:my-high-res-h265#video=h264#audio=copy#hardware
I'm personally fine with both on-the-fly transcoding and pre-transcoded solutions. Just some concerns here:
On-the-fly transcoding:
- There are still modern devices without video acceleration (e.g. Asustor AS68 series)
- Is the performance enough for smooth streaming in a reasonable resolution?
- The synchronization of high-quality stream and low-quality stream?
Since transcoding into h265 with acceleration is not supported, it's possible the transcoded h264 stream spent more time than the original h265 stream.
Pre-transcoded:
- The storage layout.
- Error handling if low quality clip is missing. (Including instance upgraded from older version)
The recording folder is already huge enough, and there are no built-in solutions to distribute them onto multiple paths.
I'm running Frigate on a RPi 5 + NVMe/Coral PCI hat and it's working great, except when reviewing events remotely over Wireguard, when I have to wait a very long time before playback starts, and often the available bandwidth is not enough to sustain the required bitrate.
For this platform, obviously, realtime playback transcoding is not an option.
Instead, it would be great to be able to record events using both the record and the detect streams, and seamlessly switch between them while reviewing events. This would be very cheap in terms CPU usage, as the extra stream wouldn't need any extra transcoding, and would only require marginal extra storage, as the sub stream is low bitrate.
I think this would be a great solution for many scenarios, where the ability to review events quickly is more important than full image quality.
If this feature was implemented, I would also add a "Default Playback Stream" setting in the UI, along with "Default Playback Rate" in Recordings Viewer settings.
It's worth mentioning that we've made some HLS optimizations so in frigate 0.16.1 playback startup will have considerably faster
It's worth mentioning that we've made some HLS optimizations so in frigate 0.16.1 playback startup will have considerably faster
I'm on dev and I've just pulled latest commit. Right now I'm connected via Wireguard over cellular network (measured bandwidth around 15Mbps, both up and down), and the time needed to play an event is 30+ seconds.
@NickM-27 Do you think you could consider my proposal?
To be clear, the change @NickM-27 referred to was not committed to dev, only to master.
I'm not sure if dev has that commit, it was made to master for 0.16.1
Using 10mbps manual browser throttle I was having 32 seconds to start playing recordings. After the fix it is now 8 seconds in that same scenario
Do you think you could consider my proposal?
Not quite sure what you mean, like it was said previously some variation of this will be done at some point. We've not pinpointed what the approach will be.
I've checked, and dev includes fb290c411b2beff0f2663414bc9de3a02720e523. Now I've switched to master and it's the same. Unfortunately I don't see any improvement in playback speed.
Not quite sure what you mean, like it was said previously some variation of this will be done at some point. We've not pinpointed what the approach will be.
I was referring to my previous post:
Instead, it would be great to be able to record events using both the record and the detect streams, and seamlessly switch between them while reviewing events. This would be very cheap in terms CPU usage, as the extra stream wouldn't need any extra transcoding, and would only require marginal extra storage, as the sub stream is low bitrate.