zstd icon indicating copy to clipboard operation
zstd copied to clipboard

Mixing skippable frames and magicless zstd frame

Open Cyan4973 opened this issue 2 years ago • 0 comments

zstd supports the concatenation of multiple frames, zstd and skippable, in a single stream. The decoder determines the boundaries of each frame, and properly decodes the complete stream.

However, in the more narrow case of magic-less zstd frames, @terrelln determined that it's possible for a magic-less zstd frame to start with a header which mimics a skippable frame magic number, introducing confusion. This scenario requires a good dose of bad luck, but it's nonetheless technically possible.

As a consequence, it might be preferable to disallow mixing of skippable frames with magic-less zstd frames. Magic-less zstd frames could be concatenated with other magic-less zstd frames, but that's all, no other type of frame would be allowed.

Note that it's already impossible to mix normal zstd frames with magic-less zstd frames, it must be only one type or the other. So there are already limits, the point would be to make them more strict, by also removing skippable frames.

It's unclear if the target scenario is employed in the wild, and as a consequence would be broken by such a change. Our guess is, probably not, because magic-less zstd frames are pretty rare to begin with, they are sometimes (rarely) employed in limited scenarios, typically with a lot of very small data, in controlled environments which tend to not mix the magic-less frame with any other data.

To be discussed for a future release.

Cyan4973 avatar Apr 04 '23 01:04 Cyan4973