Doesn't appear to produce CBR (constant bitrate) stream

[morgant]

I did a test stream today and noticed that my bitrate seemed to fluctuate a lot between 2k-4k:

I was streaming in 720p30, so fauxstream's 3500k default bitrate should be sufficient. Upon reviewing the Twitch Broadcast Guidelines I noticed that they highly suggest using CBR (constant bitrate) instead of VBR (variable bitrate):

Variable bitrate is suboptimal for video over the internet due to how TCP (the internet protocol most streaming video uses) works, and you should use CBR whenever possible.

For a 720p30 x264 stream, Twitch suggests the following settings:

Resolution: 1280x720 Bitrate: 3000 kbps Rate Control: CBR Framerate: 25 or 30 fps Keyframe Interval: 2 seconds Preset: veryfast <-> medium Profile: Main/High

Upon inspecting the fauxstream source, I see that there is a -crf option, but it's a noop and doesn't actually get used in either the video or audio ffmpeg commands.

While fauxstream does appropriately set the ffmpeg x264 -minrate & -maxrate options to the same as the bitrate, as suggested for implementing CBR, it appears to set the bitrate using a-vb option, but I believe that really should be either -b or -b:v. I couldn't find documentation of a -vb option, so it might just not be applied correctly or it might be a long-deprecated option that is harder to find.

After a fair amount of research, I found this informative Stack Overflow answer regarding changing key frame inteval and the author's "How to create ABR content with FFmpeg in one pass" blog post. I had a pretty good understanding of how ABR (adaptive bitrate) streaming over HTTP and/or TCP works, but I found this "ABR: Consistent, High-quality Streaming’s Key" [commercial] blog post to be a good refresher.

So, I believe that we should ensure we're using either ffmpeg's -b or -b:v options when setting the bitrate, plus add -x264-params no-scenecut=1. We're already setting the -g option to twice the framerate, matching Twitch's suggested keyframe ever 2 seconds, so that's good. Probably also worth verifying that we're using the 'main' profile too.

I'd also suggest we try to ensure that the AAC audio is encoded in CBR too. The default it 128k bitrate for stereo AAC, but it might be worth dropping it down to Twitch's suggested 96k bitrate.

[morgant]

Possibly related to issue #5.

[morgant]

I also found an example of turning off scene detection in ffmpeg in Twitch's 2017 Live Video Transmuxing/Transcoding: FFmpeg vs TwitchTranscoder, Part I blog post. I'll take that as confirmation we should be doing that for CBR (at least for Twitch).

[morgant]

I have a WIP branch that implements the video improvements for CBR:

• Uses -b:v ${bitrate}k instead of possibly-incorrect -vb option
• Disables automatic scene detection to ensure keyframes at 2s
• Uses the x264 'main' profile

Bitrate stayed much more consistent when locked to 3000k for 720p25, though my OpenBSD workstation was struggling to encode fast enough (too much down scaling, I believe):

I was having local audio issues, so I'm still testing/troubleshooting CBR AAC encoding. Look for a later commit.

Now that I'm setting the x264 profile, I'll add an option to override it. main is a reasonable default, but high is also an acceptable option for Twitch ingest.

[morgant]

I've switched the video codec profile from 'main' to 'high', since either are supported for Twitch ingest. I also changed the -b (video bitrate) option to -vb, though I've retained support for -b for backwards compatibility.

On the audio side of things, I've added a new -ab (audio bitrate) option, defaulting to Twitch's suggested 96k. By now always specifying the audio bitrate, the should be forcing AAC into CBR-mode (according to the documentation). I've also switched it to the fast AAC coder, in the hopes of further reducing latency.

So far this seems to be working well in my testing, but I'll continue dog-fooding.

Note: I have not added command line options for changing either the video or audio codec profiles. I can add that if anyone feels they'd use it.

[morgant]

After reviewing ffmpeg's x264 preset & tune documentation again, I compared the options set by the ultrafast preset (which fauxstream is already using) and the zerolatency tune.

From the x264 --fullhelp output:

• ultrafast (preset):

      --preset <string>       Use a preset to select encoding settings [medium]
                                  Overridden by user settings.
                                  - ultrafast:
                                    --no-8x8dct --aq-mode 0 --b-adapt 0
                                    --bframes 0 --no-cabac --no-deblock
                                    --no-mbtree --me dia --no-mixed-refs
                                    --partitions none --rc-lookahead 0 --ref 1
                                    --scenecut 0 --subme 0 --trellis 0
                                    --no-weightb --weightp 0

• zerolatency (tune):

                                  - zerolatency:
                                    --bframes 0 --force-cfr --no-mbtree
                                    --sync-lookahead 0 --sliced-threads
                                    --rc-lookahead 0

There is some overlap, but it also looks like there would be some benefit to adding -tune zerolatency as it would add the the following flags: --force-cfr --sync-lookahead 0 --sliced-threads.

[morgant]

This is the best documentation of the x264 options that I have been able to find:

http://www.chaneru.com/Roku/HLS/X264_Settings.htm

For the aforementioned options that would be added by using the zerolatency tune:

• --force-cfr:

  If using ffms2 or lavf demuxers, timecodes are copied from the input file, provided the output file is not raw. This option disables this, and forces x264 to generate its own. When using this you probably also want to set --fps.

• --sync-lookahead:

  Sets the number of frames to be used as a buffer for threaded lookahead. Maximum Value is 250. Automatically disabled during the 2nd or greater pass or when using sliced threads.

  Setting this to 0 disables threaded lookahead, which allows lower latency at the cost of reduced performance.

• --sliced-threads:

  Enables slice-based threading. This threading method produces lower quality results than the default method both compression and efficiency-wise, but adds no encoding latency.

  The maximum number of sliced threads is MIN( (height+15)/16 / 4, 128 )

Since --force-cfr mentions also setting --fps, I decided to double-check how we're specifying the fps/framerate. We're currently using ffmpeg's -r option:

Set frame rate (Hz value, fraction or abbreviation).

As an input option, ignore any timestamps stored in the file and instead generate timestamps assuming constant frame rate fps. This is not the same as the -framerate option used for some input formats like image2 or v4l2 (it used to be the same in older versions of FFmpeg). If in doubt use -framerate instead of the input option -r.

Looking at the X11 grabbing documentation, it only shows use of -framerate, so I think we should probably also switch to -framerate.

[morgant]

I've added the -tune zerolatency.

I also added -framerate, but found that I couldn't remove use of the -r framerate option or the output file would revert 29.95fps even though the actual framerate wasn't that high.

Will try to fit in a live stream tomorrow for further stress testing.

[morgant]

I have done a couple live test streams and it seems to be working, though I'm having a bit of difficulty confirming that AAC audio is actually trying to stick to a constant bitrate (CBR).

I'll be doing more testing,but I do consider this to be essentially feature-complete for my goal with this PR. So, any additional testing and feedback is welcome.

[morgant]

I was reading up further on the difference between using ffmpeg's -framerate option and the -r option (which differs in based on whether it's used in input or output; including SuperUser: How to get AVFoundation devices to have a consistent playback rate (1.0x) W/ FFMPEG on MacOS). I realized that having both -framerate & -r specified after the -i $DISPLAY$offset line in our VIDEO variable means that we're probably letting x11grab record at the display's refresh rate, then converting/encoding that down to the target frame rate.

It's probably more efficient to only pull in our target number of frames from X11, and encode it to the same target frame rate. I've tested this on older hardware (dual-core i7 2.2GHz -- boosts up to 3.1GHz -- with a 50Hz display), encoding down to 720p @ 25fps and it seems to be able to more consistently maintain the fps. So, I've merged that into this code.

---

I haven't dug further into confirming whether the AAC audio is CBR.

[morgant]

I've significantly reworked my stream configuration to remove capturing with x11grab at a higher resolution and scaling down in ffmpeg while encoding. Now I'm able to reliably capture & encode 720p25, including mpv window showing webcam, background music, and my mic. Further refinements are that I'm renice-ing fauxstream and its ffmpeg processes to a priority of -10 and nice-ing heavy applications like web browsers down to a priority of 10.

I've had several live Twitch streams with this configuration and am finding the bitrate to be consistently holding close to the 3000k I've set in fauxstream (this patchset, naturally.) Here's the Twitch Inspector summary of a 2hr stream today:

In the Twitch Stream Manager dashboard it looked like the bitrate was staying mostly flat between ~2950k and ~3050k, but you can see there is slightly more variability, but barely. So, I think I'm happy with the results I've been seeing and will submit a PR for these changes.

[rfht]

thanks @morgant, fixed/addressed now with the merged PR https://github.com/rfht/fauxstream/commit/2a489a68adeb2ac5985c2336b7f1293dc7f86632