conan icon indicating copy to clipboard operation
conan copied to clipboard

Add support for zstd compression

Open grossag opened this issue 1 year ago • 10 comments

Changelog: (Feature): Add support for zstd compression Docs: Will create one if this PR is acceptable

  • [x] Refer to the issue that supports this Pull Request.
  • [x] If the issue has missing info, explain the purpose/use case/pain/need that covers this Pull Request.
  • [x] I've read the Contributing guide.
  • [x] I've followed the PEP8 style guides for Python code.
  • [ ] I've opened another PR in the Conan docs repo to the develop branch, documenting this one.

As discussed in issue #648, this change adds zstd support to conan in the following ways:

  1. The person or build running conan upload can set a config value core.upload:compression_format = zstd to upload binaries using zstd instead of gzip.
  2. The zstd compression is done entirely in Python using a combination of tarfile and python-zstandard. Then the file is uploaded as normal.
  3. When downloading packages, if a .tar.zst file is encountered, the extraction code uses tarfile and python-zstandard to extract.
  4. Adds a test to cover zstd compression and decompression.

I chose python-zstandard as the library because that is what urllib3 uses. The package has not yet hit 1.0 but urllib3 is a mature package and it says a lot to me that they chose python-zstandard.

I apologize in advance if I'm missing important parts of the developer workflow. If this approach is acceptable, I'll create a docs PR as requested.

Developer docs on all branches say to open pull requests against develop but AFAICT that is Conan 1.x. I'm opening this against develop2 instead because that appears to be Conan 2.x; I hope that's the right thing to do.

grossag avatar Sep 08 '23 18:09 grossag