caniuse
caniuse copied to clipboard
HTTP content-encoding: zstd
Stumbled upon a relatively new content-encoding called zstd
(Zstandard).
Defined in:
https://tools.ietf.org/id/draft-kucherawy-dispatch-zstd-03.html
https://datatracker.ietf.org/doc/html/rfc8878
Repo: https://github.com/facebook/zstd
Blogpost: https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/
To clarify, it's a body content encoding. The content-coding identifier is zstd
. It's been standardized in RFC 8478. It's supported by some non-browser clients and servers (which I guess is outside your purview), and we're working towards browser support.
It's supported by some non-browser clients and servers (which I guess is outside your purview), and we're working towards browser support.
I saw that implementations, saw zstd_dict_file
option and tried to search around all internet about an example and failed. There are no public dictionaries. I think Facebook don't want to give money on that research. You should make a research yourself and create dictionary for your private application (based on production database).
So zstd is the option for private client + server applications like... facebook! So why people are creating tasks about integrating zstd into common purpose applications like browsers?
I can't understand a reason for integrating zstd even in caniuse
today.
Hi @andrew-aladev,
In general, using a dictionary with Zstd is entirely optional, you don't need one. And in fact, the HTTP extension specified in the above-linked RFC, the zstd
content-coding, does not use a dictionary. The dictionary support in the nginx extension is non-standard (see tokers/zstd-nginx-module#2 for context). So none of that is really relevant to this topic.
I'll comment on your other concerns in a reply on facebook/zstd#1669.
using a dictionary with Zstd is entirely optional
You can use zstd without dictionary or with your own dictionary.
Caniuse is related to zstd support in common purpose applications like web browsers. I think that nobody will start to integrate zstd into web browsers without clear dictionary registry. Zstd RFC doesn't declear a way of synchronizing and protecting dictionaries. It will take many years to do it.
@felixhandte
It's supported by some non-browser clients and servers (which I guess is outside your purview), and we're working towards browser support.
This confirms that it's too early for inclusion in Can I use, right?
So zstd is the option for private client + server applications like... facebook! So why people are creating tasks about integrating zstd into common purpose applications like browsers?
This is an extremely narrow way to read the situation. I use zstd at work, it has excellent technical properties that make sense for basically any user of general-purpose lossless compression. I've seen it used in the wild with HTTP APIs; yes it's internal for now, but it not being included in browsers is a matter of browsers picking up support for it... just like everything on caniuse.
This is an extremely narrow way to read the situation. I use zstd at work, it has excellent technical properties that make sense for basically any user of general-purpose lossless compression. I've seen it used in the wild with HTTP APIs; yes it's internal for now, but it not being included in browsers is a matter of browsers picking up support for it... just like everything on caniuse.
I am using it too, it is very effective in private applications where you can train dictionary based on real world data. General-purpose lossless compression using zstd without dictionary is almost meaningless.
Caniuse is related to web browsers. zstd
without dictionary is not able to compete with brotli
that uses single static dictionary optimized for web data. This question can be resolved by registration of zstd ditionaries optimized for web data in iana registries. I am waiting when facebook will make that step.
@andrew-aladev, I would hesitate to generalize a specific experience you've had into a global statement about one compressor being universally better than another. Compression is notoriously tricky that way--the specifics of the use case matter. "using zstd without dictionary is almost meaningless" is an almost meaningless statement. I can offer a counter-example: we chose zstd over brotli for HTTP traffic at Facebook.
Even without a dictionary, we found that zstd achieved equivalent compression, compared to brotli, at our target compression speed and performed much better on the client side (as a result of its much higher decompression speed). On mobile chipsets, decompression efficiency is actually very important, and that produced meaningful improvements in end-to-end / top-level metrics.
I would hesitate to generalize a specific experience you've had into a global statement about one compressor being universally better than another. Compression is notoriously tricky that way--the specifics of the use case matter. "using zstd without dictionary is almost meaningless" is an almost meaningless statement. I can offer a counter-example: we chose zstd over brotli for HTTP traffic at Facebook.
I was sure that this question will appear: I am making that research, will provide results later.
Please provide your research results based on HTTP traffic on official website. Existing results has no sense: you have re-compressed books, x-ray pictures, etc. Please provide clear results that web developer wants (we are talking about caniuse
): graphs for xhtml, graphs for css, graphs for js, graphs for fonts, etc.
Even without a dictionary, we found that zstd achieved equivalent compression, compared to brotli, at our target compression speed and performed much better on the client side (as a result of its much higher decompression speed). On mobile chipsets, decompression efficiency is actually very important, and that produced meaningful improvements in end-to-end / top-level metrics.
I've already reproduced similar results (using small amounts of web data): zstd compression ratio was significantly lower than brotli, but compression and decompression speed was faster. Now just to be clear I want to provide reproducable results.
I'll second @felixhandte 's point here, and go a little further. Decompression efficiency is extremely good with zstd.
Zstd can often produce better ratios and better decompression efficiency than lz4 at the same time, even in lz4's sweetspot. In my use case (being careful not to prematurely end frames) it is a massive difference. Far as I can tell, zstd decompresses more efficiently than any common compressor at equivalent ratios, for both my plain text and binary use cases. In my experience, zstd decompresses plain text an integer multiple of times faster than lz4 and gzip do, with dramatically better ratios; and for similar ratios between Brotli and zstd, (zstd can often do better), Brotli takes several times longer to compress. There are a lot of use cases where on-the-fly Brotli simply doesn't make sense because it will increase transfer times at any ratio that makes it worthwhile; for zstd, that list of use cases is several times smaller.
The data's not mine to give, but my two cents are.
In any event, it seems to me like the thing that should decide whether Caniuse shows something is whether any of the tracked web browsers have a release with support. Debating the merits of zstd without dictionaries is neither here nor there, once it's something available in browsers.
@xorgy, I am almost agree with you, but there is one hidden thing: absolute values.
Brotli takes several times longer to compress and decompress
We are talking about 0.002
vs 0.001
seconds for 200 KB file. This may not be reasonable for users in many cases. I am going to provide those absolute values and comparison will become more clear for everyone.
I've received first results.
Google fonts ratio
Google fonts compress performance
Google fonts decompress performance
Google fonts ratio all in one file
Google fonts compress performance all in one file
Google fonts decompress performance all in one file
- use brotli for web browser, it provides best ratio for single font.
- use zstd for mobile devices, it provides best performance with competitive ratio for single font.
- use
tar.br
ortar.zst
for fonts collection, it provides almost the same ratio.
For more information see brotli-vs-zstd.
Recommendations on serving compressed TTFs…? Isn’t any browser supporting zstd also going to support WOFF 2.0?
@charmander ttf
and otf
are just popular uncompressed web data. I am going to provide another stats for data like css
, js
, html
etc. It will be ready on the next week.
cdnjs
data differs from another collection like google fonts
: it includes all file versions (instead of last one). So results are more interesting.
Cdnjs svg ratio
Cdnjs svg compress performance
Cdnjs svg decompress performance
Cdnjs svg ratio all in one file
Cdnjs svg compress performance all in one file
Cdnjs svg decompress performance all in one file
- use svg + brotli for web browser.
- use svg + zstd for mobile device.
- use
tar.zst
for svg collection, it provides significantly better ratio thantar.br
.
So we can see that zstd
is much (40-60%) better than brotli
for fonts or svg collections in one archive. Brotli is only competitive for collections when it consist from completely unique files.
Cdnjs css ratio
Cdnjs css compress performance
Cdnjs css decompress performance
Cdnjs css ratio all in one file
Cdnjs css compress performance all in one file
Cdnjs css decompress performance all in one file
- use css + brotli for web browser.
- use css + zstd for mobile device.
- use
tar.zst
for css collection, it provides huge100-130%
ratio gain overtar.br
.
Now you can see visually what I was talking about: zstd
is significantly better than brotli
for large web data collection: 1 GB +
. But brotli
is better for single small file. It means that zstd
suffers from lack of dictionary.
Results for js
files will be ready on the next week.
I appreciate the data. :+ ) I just have one thing to add.
- use svg + brotli for web browser.
- use svg + zstd for mobile device.
Some people use web browsers on mobile devices, lately.
Sure, today we have powerful smartphones with almost desktop browser. But I think many developers still optimizes applications for old phones so zstd
will be usefull.
I've underestimated the power of cdnjs
collection: it has about 220 GB
of js
files. I've received only charts for minified js files. Charts for non-minified js and for all js files will be ready on the next week.
freedom of speach smart adse
Sent from my iPhone
On 30 Jun 2020, at 1:25 am, Aaron Muir Hamilton [email protected] wrote:
I'll second @felixhandte 's point here. Decompression efficiency is extremely good with zstd.
Zstd can often produce better ratios and better decompression efficiency than lz4 at the same time, even in lz4's sweetspot. In my use case (being careful not to prematurely end frames) it is a massive difference. Far as I can tell, zstd decompresses more efficiently than any common compressor at equivalent ratios, for both my plain text and binary use cases. In my experience, zstd decompresses plain text an integer multiple of times faster than lz4 and gzip do, with dramatically better ratios; and for similar ratios between Brotli and zstd, (zstd can often do better), Brotli takes several times longer to compress. There are a lot of use cases where on-the-fly Brotli simply doesn't make sense because it will increase transfer times at any ratio that makes it worthwhile; for zstd, that list of use cases is several times smaller.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.
Unfortunately today we have received ac power loss and my ups with 100ah battery was not able to handle 5h offline. I am going to provide results later, sorry.
Unfortunately today we have received ac power loss and my ups with 100ah battery was not able to handle 5h offline. I am going to provide results later, sorry.
If you'd like, I can lend you a shell on a Ryzen 3950X with 64GiB of DDR4 in New York which has a stable connection and lives in a rack with backup power, then you don't need to run those from home.
Thank you, everything will be all right.
Cdnjs js ratio
Cdnjs js compress performance
Cdnjs js decompress performance
Cdnjs js ratio all in one file
Cdnjs js compress performance all in one file
Cdnjs js decompress performance all in one file
Wikipedia html ratio
Wikipedia html compress performance
Wikipedia html decompress performance
Wikipedia html ratio all in one file
Wikipedia html compress performance all in one file
Wikipedia html decompress performance all in one file
We can see absolutely the same result for js
and html
files.
@felixhandte We can see that zstd
provides better ratio than brotli
only when source is large enough. brotli
is better than zstd
when source is small (regular web browser usage). It proves that zstd
suffers from lack of dictionary.
Please mark public dictionary optimized for web content as priority milestone.
Please mark public dictionary optimized for web content as priority milestone.
Is there a ticket for that? I can't find one but I'm not sure.
So what's the status here? Just coming from https://calendar.perfplanet.com/2021/
- Wikipedia: https://en.wikipedia.org/wiki/Zstd
- Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=1246971
- Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=zstd
Zstandard browser support test https://www.daniel.priv.no/tools/zstd-browser-test/
Intent to Prototype by Chrome:
- https://groups.google.com/a/chromium.org/g/blink-dev/c/GDsI0Hw-jYk/m/AsQS-km-DQAJ
- https://chromestatus.com/feature/6186023867908096
Now available at https://caniuse.com/zstd - Supported behind a flag in Chrome Canary