Jetty Brotli compression support for on-the-fly compression?

[t1707]

https://github.com/eclipse/jetty.project/issues/288 There is a closed issue regarding brolit compression support but arguments justified for closing the issue are not 100% true.

According to the latest comparison blog (posted by Akamai) regarding brotli and gzip compression performance, brotli-4 is considerably better than gzip-6 in both the aspects(speed and compression ratio).

Quote directly taken from blog posted by Akamai

As you can see from the results, if you tweak the Brotli settings a little bit, the compression speeds can actually surpass the speeds of GZIP while still providing some significant savings. Based on these 1000 HTML files, running Brotli at a setting of "4" would not only save more bytes than GZIP's default setting, but it would compress about 21 percent faster as well. That same setting--"4"--showed up consistently as the break-even point when tested on individual files as well.

Another quote on dynamic aspects

Though Brotli slows down the higher the compression setting, it's entirely possible to trade-in some of those bytes for a faster compression setting. If you're going to replace GZIP for compression of dynamic assets, a Brotli quality of "4" is a good place to start. If you're running it ahead of time and queuing up those resources for a later request, then run it full blast and watch the bytes fade away.

https://blogs.akamai.com/2016/02/understanding-brotlis-potential.html

[joakime]

Biggest issue is the lack of a performant Java implementation of brotli.

[joakime]

While decompression is reported to be better, the encoding is still very slow. Dynamic encoding/compression with brotli is still something that requires more improvements/experimentation before it will be widely adopted.

That's what Issue #288 is all about, supporting the precompressed brotli features that most other brotli supporting servers also support.

[nixxcode]

Hi,

Just wanted to weigh in with this: https://jvmbrotli.com/

It's a Brotli library for Java that I created while implementing Brotli compression capabilities for https://github.com/tipsy/javalin

The compressor/decompressor Java code is taken directly from the google/brotli repo, so performance should be identical.

I haven't done any in-depth performance benchmarks yet, so these are only first impressions for now and shouldn't be taken as fact: Quality setting 4 appears to have performance comparable to the Java default Gzip, with 5-7% improved compression (compared to uncompressed) on average.

If you do decide to experiment with it, feedback is always appreciated.

[nixxcode]

Also, I realize this is a really old issue. Not sure if I should have created a new one or if posting this here was the correct thing to do.

Please do let me know if you even see this. (You being someone on the Jetty team)

[gregw]

We see this and thanks for the link. So if/when we implement dynamic brotli we will definitely look at this.... just this is not currently a high priority for us at the moment. But very happy to accept PRs for this. If anybody does work on this, I would target only jetty-10.

I note that chrome request have Accept-encoding :gzip, deflate, br, so even if we offerred brotli it would not commonly be used until browsers start requesting br first... and we are not about to disable gzip any time soon.

[maffe]

I note that chrome request have Accept-encoding :gzip, deflate, br, so even if we offerred brotli it would not commonly be used until browsers start requesting br first... and we are not about to disable gzip any time soon.

Looking at RFC 7231 section 5.3, the order seems to be insignificant, all encodings have the same default weight of 1, so a server would be free to choose any of those and Jetty may serve Brotli in this case.

In fact, website operators might want to serve Brotli in order to reduce traffic even when other encodings are preferred by the client (as long as Brotli is supported), however I don’t know if Jetty can be configured this way.

[nixxcode]

Looking at RFC 7231 section 5.3, the order seems to be insignificant, all encodings have the same default weight of 1, so a server would be free to choose any of those and Jetty may serve Brotli in this case.

This is what we do in Javalin. Brotli is optional, but if it's enabled then it gets prioritized (if the client supports it)

[gregw]

I agree we are free to pick a an encoding of equal preference... but still a bit more complex code that contains("gzip").

As I said, we are ok to have this, just not yet a prority for us.

[stale[bot]]

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.

[cowwoc]

Was this issue resolved by https://github.com/eclipse/jetty.project/issues/288? If not, what is the difference between the two?

[ReillyBrogan]

Was this issue resolved by #288? If not, what is the difference between the two?

It was not because that only covers serving pre-compressed files. This issue is for integrating a brotli encoder into Jetty such that it's possible to serve content that was not already compressed with brotli by the time it got to Jetty (for instance compressing JSON responses from your web application running with Jetty as the HTTP server).

[cowwoc]

Makes sense.

How do we remove the stale label?

[gregw]

stale label removed. Still a low priority and may only get addressed in jetty-12

[joakime]

Before Jetty can use the jvm-brotli project (mentioned by @nixxcode) the license needs to be properly identified.

I opened an issue there for the licensing issue https://github.com/nixxcode/jvm-brotli/issues/44

Once that is addressed, we can submit a request for use to Eclipse IP (known as a CQ) and see if the license and code vetting allow Jetty to use it.

[tipsy]

@joakime just a heads up, @nixxcode has seemingly disappeared from the internet, the project can probably be considered abandoned. On the bright side, it's been included in Javalin for 3 years, with no (reported) issues.

[joakime]

@tipsy shame, as we would be force to remove it from Jetty as it's abandonware due to various corporate policies around this kind of thing.

Is there a fork of the project that's better maintained somewhere?

[tipsy]

This appears to be the only fork ahead of @nixxcode: https://github.com/SiggiIO/jvm-brotli. Are you interested in releasing and maintaining your fork @SiggiIO? I worked with @nixxcode when he implemented it, and it didn't seem like too much work (it's mainly wrapping the Google library). He also appears to have done a good job documenting everything, so it shouldn't be a big effort to maintain.

[WesternIcelander]

My fork only has very minor changes to make jvm-brotli work on Apple Silicon Macs running natively and not using Rosetta. I don't mind if anyone wants to include my commits in their own fork, but I'm not able to maintain or provide support for my fork at this time.

[joschi]

A quick search on Maven Central brought up this library, which could be used for this feature and looks well maintained: https://github.com/hyperxpro/Brotli4j

It also seems to be used by vert.x: https://github.com/eclipse-vertx/vert.x/pull/4262

[tipsy]

Inspired by @joschi's comment I did a little digging too, and it seems the official Google release is https://mvnrepository.com/artifact/org.brotli/dec/0.1.2, which is used by 60 artifacts (https://issues.sonatype.org/browse/OSSRH-25741?jql=text%20~%20%22brotli.org%22).

[joakime]

The google brotli lib is only at 0.1.2 on maven central.

https://search.maven.org/artifact/org.brotli/dec

And hasn't seen a release since May 2017

Meanwhile, the main codebase still chugs on, all the way up to version 1.0.9

https://github.com/google/brotli

Both the Brotli4j and Google projects rely on JNI to native libs to function. This will complicate things for us as both projects have extremely limited scope of supported architectures. (linux 64-bit, linux arm 64-bit, windows 64-bit, and mac os catalina 10.15)

The windows install cannot be done automatically, as it requires a specific VC++ redistributable to be installed. The linux builds are dynamic builds, not static, which means we would have to recompile them to be used in about 40% of our docker images.

Brotli4j only licenses their java code and jar binaries. Their native code is a mix of unspecified license (no license statement in native code), with some MIT licensed from google. Brotli4j says they are Apache 2.0 Licensed. But they seem to use Google native code, that doesn't match up actual google native code (did they modify it? use it as is?), that would mean the native libs are MIT licensed if unmodified. But why copy the google native code into their repo? it exists already on github, link it, use it as-is, use released versions to build from. What version of the google native libs are they using? Kind of important for security tracking / advisories / etc.

[hyperxpro]

Their native code is a mix of unspecified license (no license statement in native code), with some MIT licensed from google. Brotli4j says they are Apache 2.0 Licensed.

Classes that are modified have dual-license which are Apache 2.0 License and MIT License. Classes that are as-it-is only have Google Brotli MIT License.

But they seem to use Google native code, that doesn't match up actual google native code (did they modify it? use it as is?), that would mean the native libs are MIT licensed if unmodified.

As mentioned above, Brotli4j contains both modified and unmodified code.

But why copy the google native code into their repo? it exists already on github, link it, use it as-is, use released versions to build from.

There are modifications to make the library work like changing JNI method names.

What version of the google native libs are they using? Kind of important for security tracking / advisories / etc.

Currently, Brotli4j tracks the latest Brotli commit from the Google Brotli repository. I will add tracking in near future to makes easier for others to find which revision of Brotli code is used.

[github-actions[bot]]

This issue has been automatically marked as stale because it has been a full year without activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gregw]

Making this unstale. It is something we really should do... just has not got to the top of the priority queue yet.

[joakime]

Tip: the label Pinned will prevent issues from going stale.

[cowwoc]

What's interesting is that per https://caniuse.com/?search=gzip Brotli is supported by 95% of browsers vs 85% for gzip.

[joakime]

The details on caniuse.com are the other way around for the fundamental Content-Encoding behaviors.

100% for gzip 95.2% for brotli

[gregw]

We could easily do a BrotliHandler as a copy of Gzip. The hard part is to do either/or in one handler neatly.

[joakime]

The other hard part is the Brotli Java support. Which, last time we investigated this we hit the following issues.

• Wasn't 100% java (it had required native libs)
• The experimental 100% java versions were horribly inefficient, to the point were it was faster to just use uncompressed content.
• Wasn't mature, (eg: only supported only decompression, not compression, or didn't support Java streams of channels, or only worked as a command line tool)
• Had a license that we cannot use.