etcher
etcher copied to clipboard
balena-io
Add zstd support
Thank you for your request. As @lurch pointed out it was previously discussed about a year ago, so we can at least see if the situation got better wrt dealing with 7-zip
@sedlund Out of curiosity, are you aware of disk images that are being distributed in zstd format?
@lurch I haven't seen anything offered as of yet - I suppose its a bit chicken and egg now.
I use it for creating SD backup images compressed on the device and then streamed over the network to its final destination. I'm sure you've seen the benchmarks, Zstd is far and away faster in compressing on small devices,
7zip itself doesn't have Zstd support. There is a fork that includes it: https://github.com/mcmilk/7-Zip-zstd/ There are also some good benchmark graphs showing the advantages, although using an older Zstd 1.2 - it has received many large performance increases since then.
I'm not sure why you'd need 7zip to implement Zstd though. I would think it would be implemented similarly to how lzma was in Etcher.
I would think it would be implemented similarly to how lzma was in Etcher.
A quick search finds https://www.npmjs.com/package/zstd which was last published 4 years ago and https://www.npmjs.com/package/node-zstandard which was last published 3 years ago :slightly_frowning_face: More chicken & egg? :chicken: :egg:
@lurch I guess a lot of iteration is not required when you only need to implement 2 methods to hook to an external lib - compress and decompress eh? 😃
Sorry, I was indirectly referring to your "using an older Zstd 1.2 - it has received many large performance increases since then" comment - have these "performance increases" been added before or after these node packages were last published? :man_shrugging:
I've been looking into this because of the speed of ZSTD's decompression, combined with its good compression ratio, which I think offers a uniquely useful algorithm for compressing disk images and the like.
It looks like the main blocker is the age of existing ZSTD node modules, the lack of a module that implements a node Transform stream, and the small Venn diagram of people who know Node and C++ well enough to make a native module implementing this interface.
Fortunately, compiling libzstd to WebAssembly is pretty straightforward with Emscripten, at which point we can just call exported methods directly from JavaScript. I'm no JS guru, but I know my way around Emscripten, and this is an interesting project to me, so I guess we'll see where it goes.
Is there any news on this topic? I would also require to flash .zst compressed images
I am producing .img.zstd images for Armbian, and would push for that to be standard if Etcher supported it.
Really large images, like operating systems, are starting to use zstd more. That's what brought me here. https://github.com/starfive-tech/Fedora_on_StarFive This 3.5GB zstd binary expands to 13G. It would be 4.6GB as a gzip and it's slower to decompress.
So to @lurch's question "are you aware of disk images that are being distributed in zstd?" the increasing answer is yes and interest seems to be trending up. I hadn't heard of ZSTD before a year ago, so I don't perceive this as a red hot issue, but it must be growing.
In my case, I just wanted a streaming decompress so I don't have to read the big file to write a bigger file and then use Etcher to safely copy that. I'd be perfectly happy for that 'bigger' image to exist only in temporary memory as it was streamed to the USB card.
If libzstd doesn't float your boat, cloudflare has cleaned up the really really old zlib code, branched it, and added zstd. See https://indico.cern.ch/event/695984/contributions/2872933/attachments/1590457/2516802/ZSTD_and_ZLIB_Updates_-_January_20186.pdf
While I'm here, thank you for a great cross-platform utility. I know those are hard to make and Etcher rocks.
Chiming in here - I use zstd heavily in my image workflow for SBC distro and ChromeOS images. Zstd support would save me a lot of time, xz is space efficient but has crappy compress/decompress compute and time efficiency.
[mpous] This has attached https://jel.ly.fish/d5aa35db-a306-4791-b408-4817d247a383
I agree. Since #2048 was closed as uncompleted, zstd is another good option for those who don't want the long wait times and high resource consumption of xz
Good timing ping. With the recent xz security fiasco, many distros are reevaluating or switching to zstd, such as OpenWrt.