Support zstd as first-class compression scheme

[nacl]

https://github.com/facebook/zstd/ is another compression format with highly tunable compression format that highly desirable performance characteristics. It would be nice if it were supported locally without having to specify an external compressor, when we are able to easily do so.

Notes that come to mind:

• zstd in python3 requires external library bindings
• On top of that zstd in zip files is not immediately supported without changes to the standard library
• pkg_tar (and thus pkg_deb) can rely on an external compressor
• pkg_rpm can compress with zstd by passing binary_payload_compression = "w$LEVEL.zstdio, where $LEVEL is a numeric compression level. This requires zstd support in the local rpmbuild binary.
• pkg_deb is kinda up in the air: the Debian package format does not appear to support zstd yet, but apparently ubuntu is already using it. See also https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-21.10-Zstd-Debs, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664.

[adam-azarchs]

If it helps, I have a BUILD file to build zstd from source that we use extensively in our rule set, which I could contribute here. No promisses about how well it works outside of linux/x64, but it was basically transliterated from their BUCK build file. For our own use cases we just pass that executable as the compressor executable for pkg_tar.

[aiuto]

It would be more useful to release it to a distinct repo so other rules can use it. Or.... and this would probably be difficult ... upstream it to the zstd repo.

[adam-azarchs]

They might accept it? https://github.com/facebook/zstd/issues/3123

[dzbarsky]

@adam-azarchs mind sharing your build file/example if you have it handy?

[adam-azarchs]

Sure, attached here BUILD.zstd.bazel. Note:

1. I didn't put any effort into checking whether it works on platforms besides linux/x64
2. I didn't enable support for features I don't need (like zlib support in the zstd executable, or support for "legacy" mode).
3. It's using our own internal fork of rules_license which has some additional fields which aren't supported by the main branch.