sdk-ng
sdk-ng copied to clipboard
zephyrproject-rtos
Use LZMA for distribution bundles
From https://github.com/zephyrproject-rtos/sdk-ng/pull/287#issuecomment-1265353916:
GitHub does not allow uploading release assets larger than 2GB and we will not be able to upload the "all-in-one" distribution bundles to the GitHub releases with the picolibc also bundled in the toolchains.
We can resolve this in one of the following ways:
- Upload the "all-in-one" distribution bundles to a different file hosting service (e.g. S3).
- While this would be the simplest solution, it will incur a lot of extra cost in terms of outbound traffic.
- With the size of the distribution bundles and the number of downloads, this can be in the range of thousands of dollars per month.
- Use a better compression algorithm
- Zephyr SDK currently uses GZip (tar.gz) for Linux and macOS, and DEFLATE (zip) for Windows.
- GZip and DEFLATE are not exactly known for having the best data compression ratio.
- Using LZMA should give much better compression ratio (a quick test has shown around 1.5x for the "all-in-one" distribution bundle with xz-utils).
- Use XZ (tar.xz) for Linux and macOS, and 7z/LZMA (7z) for Windows. This should keep the "all-in-one" distribution bundle sizes to below 2GB.
- LZMA requires more processing power and takes longer to compress and decompress, but this is not so much as to prevent its adoption (a quick test has shown that decompressing "all-in-one" distribution bundle tar.gz took approx. 1 minute whereas tar.xz took approx. 2.5 minutes on my machine, which is not that bad).
- Of course, even after switching to LZMA, future changes may increase the distribution bundle sizes to more than 2GB, in which case we will have to evaluate the multi-volume archive approach described below.
- Use multi-volume archives
- As its name states. Split single compressed archive into multiple volumes/parts, which can be combined together after download and decompressed.
Switching to LZMA gives us a temporary reprieve and doesn't change workflow for users, so it's a pretty simple choice to make. Moving forward, it's pretty clear we'll need a solution that breaks the image into pieces at some point, that could also allow inclusion of debug symbols if the pieces are small enough. I think that would require writing a tool to help developers manage the SDK installation?
I think that would require writing a tool to help developers manage the SDK installation?
https://github.com/zephyrproject-rtos/zephyr/issues/37255 is basically where we want to be, eventually.
Merged to main as part of https://github.com/zephyrproject-rtos/sdk-ng/pull/602