zstd
zstd copied to clipboard
facebook
Ways to allow projects to use a minimal decompression-nodict implementation
2 attempts now where made to bring zstd into busybox and there are open questions on whether it is realistical to keep in sync with upstream, since the aim is to bastardize alot in favor of small code-size. (I only agree to extend, as one of zstd characteristics is speed)
What I of course would want is an as painless as possible way to compare with current upstream, so my plan would be to
- Forward all acceptable changes upstream (first try: https://github.com/facebook/zstd/pull/2805)
- Discuss how more controversial things that uglify/break code could be handled (upstream-unable part of a change: https://github.com/nolange/zstd/tree/optional_dictionary_support)
- try to leverage the existing
freestanding.py, for example with a functionality that takes a list of names and cuts out the declaration/definition of functions or structs matching a name in the list
References: busybox Branch for zstd: https://github.com/nolange/busybox/tree/zstdapplets mailing list discussion of (another) zstd patch: http://lists.busybox.net/pipermail/busybox/2021-September/089179.html
If your goal is minimal binary size as a priority, there are additional build flags that could prove useful described in this documentation.
If your goal is minimal binary size as a priority, there are additional build flags that could prove useful described in this documentation.
That is merely a starting point ;) I 'd recommend taking a peek at https://github.com/nolange/busybox/tree/zstdapplets/archival/libarchive/zstd . Most invasive change is ripping out anything related to dictionaries.
Hi @nolange,
I'm really excited to see your interest in bringing zstd to busybox! We'd be very happy to be included in the toolkit.
It sounds like the overriding need for busybox is to minimize code size. As you may have noticed, this is not the situation that our implementation primarily targets. :smile: The overwhelming priority for most of our users is getting the most performance possible.
But we certainly recognize that different use cases have different needs, and to the extent that we can support less common use cases in the reference zstd implementation, we'd like to. So if practical, we'd be happy to host the necessary changes in the mainline. That way we can get a test set up for your use case in our continuous-testing infrastructure, so we make sure future changes work for you.
And it would be nice to do this in part because busybox is also not the only use case where binary size is really important! We've had a number of folks wanting to do zstd decompression in the browser via WASM, and lib size is an important optimization axis there as well. So this work would benefit them as well.
The question is just how to serve this different profile in a way that co-exists with the main performance-optimized zstd.
So far it looks like everything you're doing is introducing preprocessor conditionals. This is great in terms of knowing that when not activated, it won't have any impact at all on the default configuration. The cost is just codebase complexity. But hopefully we can keep that not too crazy. And good contbuild coverage will help. #2805 seems fine to me. Even b40046fb707b96b4a08c79321586612ee74a37f2 isn't toooo terrible to my eyes, although I have some questions.
What do you imagine the total scope of changes would be that you would want to make?
Hi @nolange,
I'm really excited to see your interest in bringing zstd to busybox! We'd be very happy to be included in the toolkit.
Just to avoid misunderstanding, I am as much of a busybox maintainer as I am for zstd ;)
It sounds like the overriding need for busybox is to minimize code size. As you may have noticed, this is not the situation that our implementation primarily targets. The overwhelming priority for most of our users is getting the most performance possible.
Yeah, Id like to see busybox having both a minimal implementation to load a few compressed kernel modules from an initramfs, as well as something more "beefy" for a running system.
The question is just how to serve this different profile in a way that co-exists with the main performance-optimized zstd.
So far it looks like everything you're doing is introducing preprocessor conditionals. This is great in terms of knowing that when not activated, it won't have any impact at all on the default configuration. The cost is just codebase complexity. But hopefully we can keep that not too crazy. And good contbuild coverage will help. #2805 seems fine to me. Even b40046f isn't toooo terrible to my eyes, although I have some questions.
What do you imagine the total scope of changes would be that you would want to make?
My plan is pretty much outlined in https://github.com/nolange/busybox/commits/zstdapplets, export from upstream is ecplained in archival/libarchive/zstd/README.source, it comes with a config option to compromise between size or speed. I don't want any sweeping changes before I know which way busybox wants to go. I brought down the minimal cost to around 17KB, but I have no idea how small some hypothetical minimal zstd decoder would be.
I see the benefit of having upstream maintaining and regression testing the used codepaths, hope that's the way forward.
(Was that all the questions?)
I think so. :)
So it sounds like there are a set of code changes we could merge, and the rest would be a mechanical conversion that we could automate (and again, run in continuous testing). That works for me!
I guess keep us updated about what the busybox folks say?
@nolange, I'm curious if you have any updates on this topic. We've been talking to the Chrome folks about integrating Zstd into the browser and they also care a lot about binary size (although maybe that ship has already sailed.....). So I'd like to see this topic move forward. Do you intend to keep working on it?
Unfortunately, no news. gonna prod the ML again. I am gonna work on it as far as getting it into Busybox (unless that requires writing a minimized zstd from scratch).