updater: Use Zstandard for delta update and file compression

[derrod]

Description

Adds support for delta patches created with Zstandard rather than bsdiff.

ToDo:

• [x] Add support to bouf
  • Supported added in v0.4.0
• [x] Add Zstandard to obs-deps
  • PR opened: https://github.com/obsproject/obs-deps/pull/150

Motivation and Context

• Faster patch generation (nearly 2x at level 22, 4x at 19 which offers comparable size to bsdiff)
• Smaller patches (~10 MiB less for 27.2.4 => 29.0)
• Lower memory usage on client (TBC)

How Has This Been Tested?

In-dev bouf branch and local update server setup.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• [x] My code has been run through clang-format.
• [x] I have read the contributing document.
• [x] My code is not on the master branch.
• [x] The code has been tested.
• [x] All commit messages are properly formatted and commits squashed where appropriate.
• [x] I have included updates to all appropriate documentation.

[derrod]

With the last two comments I switched from using zlib to using zstandard for compressing the JSON body of the patch manifest request and removed zlib entirely.

The manual gzip decompression has been removed as WinHTTP supports doing this for you as of Windows 8.1 (via WINHTTP_OPTION_DECOMPRESSION).

[derrod]

Note for CMake stuff: zstd outputs zstdConfig.cmake and zstdTargets.cmake which become part of obs-deps once that PR is merged. When I was building locally CMake was able to automatically pick those up and building with zstd simply required adding the find_package(zstd) and the static target to CMakeLists.txt

Edit: Since ENABLE_UPDATER is on by default the build currently fails as zstd is not yet part of obs-deps.

[derrod]

Here's a branch that additionally removes LZMA as well (both from deps/ and the updater): https://github.com/derrod/obs-studio/compare/updater-zstd..updater-remove-lzma

If we ever want to feel better about the number of lines of code in our repo that's an easy -27,000 :P

[notr1ch]

What do you think about sending Accept-Encoding: zstd and decompression of Content-Encoding: zstd for non-patch files? Basically we'd be implementing our own decompression again like we previously had for gzip, but using zstd which should achieve speed and compression wins.

[derrod]

I've thought about using compression for non-patch files, but I didn't consider of doing it as part of the download itself. That seems fine though.

I assume we'd want to do ahead-of-time compression there as well. I'd just need to know what that would mean for bouf as far as creating the directory structure goes. For example could I just output both file.ext and file.ext.zst into the directory and nginx could automatically rewrite the download to the zst file if the client advertises support?

[notr1ch]

That should work. Technically we shouldn't even need to keep the uncompressed versions around but I suspect there will be users behind corporate MITM proxies that strip unknown content-encodings etc. Maybe it's better to just download the .zst directly and decompress outside of the HTTP connection.

[derrod]

Now updated with support for compressed non-patch files. Support for building compatible update files and manifests was added in bouf v0.4.1.

Using compressed update files brings the download size for OBS 29.0 down from ~352 MiB to ~114 MiB. Zstandard delta patches in my test case (27.2.4 => 29.0.0) would come in at around ~65 MiB - roughly 8 MiB less than the old bsdiff + lzma based deltas.

The implementation here assumes files are compressed if the compressed_hash in the file object inside the manifest is set. That hash is used to verify the downloaded file before decompression, which happens in the download worker threads.

[derrod]

Comparison of speed of this PR with #8115 and #8106 also applied: This is running with patches served from my NAS so there's some lower latency of course, but even when serving files from a real world setup (Cloudflare R2) it did shave off another 3-ish seconds off, bringing us to around 5 seconds for 28.1.2 => 29.0.0!

[derrod]

~~Conflicted due to #8115 now being merged, will wait #8142/#8141 to be merged and obs-deps update with zstd to be released before rebasing and fixing it.~~

With #8141 merged and #8142 not conflicting I've now rebased this upon latest master.