Support Brotli as an encoding scheme when client has Accept-Encoding

[jmarantz]

See https://code.google.com/p/chromium/issues/detail?id=452335 which tracks Chrome support for Brotli in https.

Mozilla has already added support: https://bugzilla.mozilla.org/show_bug.cgi?id=366559

[igrigorik]

:+1:

[jeffkaufman]

Current plan for this:

• @crowell is adding support for caching encoded resources, starting with gzip
• once there's good browser support we'll look at extending that to include brotli compression

[crowell]

caching gzip-encoded resources is in with 1.10.33.0 and newer brotli support is started here https://github.com/pagespeed/mod_pagespeed/blob/master/pagespeed/kernel/util/brotli_inflater.h but after profiling, the current version of brotli (0.3.0) is too memory intensive in encoding to comfortably deploy. this will be revisited in the future when brotli becomes more efficient at encoding.

[jeffkaufman]

Brotli resource usage at level 10 should now be low enough that we'd be able to do this.

On the other hand, @crowell's initial investigation suggests this is more work to integrate than we had thought (on our end), so we've needed to bump the priority down here some.

[quixote911]

Brotli 0.4.0 is out - https://github.com/google/brotli/releases

Any plans to re-evaluate how 0.4.0 does with mod_pagespeed?

[jmarantz]

Based on the numbers we've seen, Brotli is now in good enough shape we can deploy it. However we are pretty tight on human resources now and it is feasible but somewhat complex for us to cache alternate encodings for different browsers.

We definitely want to do it and believe it will help by reducing network bandwidth by up to 20% for CSS and JS. It simply competes with other optimizations and bug fixes we want to do.

On Thursday, June 23, 2016, Ankit Arora [email protected] wrote:

Brotli 0.4.0 is out - https://github.com/google/brotli/releases

Any plans to re-evaluate how 0.4.0 does with mod_pagespeed?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pagespeed/mod_pagespeed/issues/1148#issuecomment-228010234, or mute the thread https://github.com/notifications/unsubscribe/AB2kPd9cW8VNf1CPqrrMiFi0PFG17WXgks5qOl9agaJpZM4GHrKA .

[AWOL-TECH]

Hello,

Just wondering if Brotli support was ever managed? Would love to be able to serve brotli compressed files over gzip :)

[GuillaumeRossolini]

@timeassistant in case you are interested in a fallback, recent versions of httpd support Brotli out of the box (v2.4.26 and up).

[AWOL-TECH]

@GuillaumeRossolini thanks, I already have the brotli module enabled. I was just curious if pagespeed was able to use brotli aswell

[tomayac]

While we wait for native support (🙏 thanks in advance to everyone involved in this effort), I've just documented how to make the current PageSpeed work with Brotli.

[iCounsellorUK]

While we wait for native support [...] I've just documented how to make the current PageSpeed work with Brotli.

BRILLIANT!

Found that page today, understood what the instructions were meant to achieve (always helpful for developers) and put it in place. Confirmed working (tested with Chrome & Firefox - looking for "content-encoding: br" in Response Headers for .css, .js etc).

Such a freaking awesome discovery - thank you @tomayac

[Sir-Will]

While we wait for native support (🙏 thanks in advance to everyone involved in this effort), I've just documented how to make the current PageSpeed work with Brotli.

Doesn't seem to be working in nginx. (pagespeed HttpCacheCompressionLevel 0;) It randomly picks gzip or br, even if only the image filter is enabled not all js files are br.

[Lofesa]

Hi Brotli is out of the pagespeed scope, it is enterely nginx config. What mime types have you enabled i the nginx brotli config (brotli_types directive)?

[Sir-Will]

I'm using the recommended settings:

brotli on;
brotli_comp_level 6;
brotli_static on;
brotli_types application/atom+xml application/javascript application/json application/rss+xml
             application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype
             application/x-font-ttf application/x-javascript application/xhtml+xml application/xml
             font/eot font/opentype font/otf font/truetype image/svg+xml image/vnd.microsoft.icon
             image/x-icon image/x-win-bitmap text/css text/javascript text/plain text/xml;

I was also unable to set the Cache-Control header via add_header and more_set_headers for js files when pagespeed is enabled with only the image filter active. ngx_pagespeed seems to completely break/prevent header modifications from other modules or settings in the nginx config.

[Lofesa]

Hi

Sound strange. PageSpeed has nothing to do with the brotli module or even gzip. Only if HttpCacheCompressionLevel is enabled it stores the optimized resource gziped and serve it compressed, if set to 0 the optimized resource is stored w/o compressing it and served as it is, then nginx compression modules compress the optimized resource.

Yes, header manipulation is an old question related on how nginx work (the order nginx processes modules).See. https://github.com/apache/incubator-pagespeed-ngx/pull/906 and https://github.com/apache/incubator-pagespeed-ngx/issues/1005#issuecomment-160901857 and you can use pagespeed AddResourceHeader "X-Foo" "Ba\nr"; too

If you only enables image filters.... well there is a PageSpeed function that work when PageSpeed don´t fully optimize resources: IPRO (https://www.modpagespeed.com/doc/system#ipro), an on the fly optimization, and there is an other side that maybe considered, the rewrite level, by default (if you don´t set any other rewrite level) is corefilters, this enables a set of filters. See: https://www.modpagespeed.com/doc/config_filters.html#level.

If you only need these filters you will and none of the others filters you need to set:

pagespeed RewriteLevel PassThrough;

This only enable the filters you have explicitly set. And maybe you need some like:

pagespeed ForbidAllDisabledFilters true;

to not permit the use of filters not explicitly enabled.

P.S. Can you share any url to test it?

[Sir-Will]

Hm, I'm no longer able to reproduce the issue with the compression, maybe a cache related issue which cleared out today?

Though I'm still facing the issue with not being able to set headers. I have pagespeed RewriteLevel PassThrough; set, so there are no modules loaded.

If you only enables image filters.... well there is a PageSpeed function that work when PageSpeed don´t fully optimize resources

I'm mainly focused on the convert_jpeg_to_webp filter (which InPlaceResourceOptimization doesn't do), though the other default filters wouldn't hurt.

I did rebuild the module with the POSITION_AUX=true environment variable yesterday, though after replacing the module there was no change. Not sure if I did build it wrong, or it's not working with dynamic modules.

One workaround is to disable pagespeed before adding the headers, but that’s not the best solution. In this case below it works out because it's only being applied to none pagespeed files.

location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
	add_header "" "";
}
location ~* \.(css|gif|ico|jpe?g|js|png|svg|json)$ {
	pagespeed off;
	expires 30d;
	add_header Pragma public;
	add_header Cache-Control "public";
}

[Lofesa]

Yes, maybe a cache issue. If you change HttpCacheCompressionLevel to 0 and don´t clean the pagespeed cache, by default this level is set to 9 (if I remember), so optimized resources are stored gziped.

But this, InPlaceResourceOptimization, can change the headers.

Me too, not sure if the role of POSITION_AUX=true is still valid with dynamic modules, but I have mentioned it to say is an issue related on how the order nginx modules works is related to how headers are modified.

W/o a whole knowlwdge of your installation I cna´t help much more, but I can suggest some things: As far as I can understand you only will transform jpg to webp images and let css and js files as it are. Why not disallow css and js files, some like:

pagespeed Disallow */*.css*
pagespeed Disallow */*.js*

or use LoadFromFile (See: https://www.modpagespeed.com/doc/domains#ModPagespeedLoadFromFile)

[jmarantz]

I don't think you should need to flush the pagespeed cache when changing options. A hash of all the options are incorporated into the cache keys.

However I think it might be that PageSpeed is overriding the brotli module and serving its own cached response. I am not sure how the pagespeed resource serving prioritizies against the nginx brotli module in the serving flow.

[Lofesa]

Hi @jmarantz When HttpCacheCompressionLevel has a non zero as value, the optimized resources are served gziped, no matter if you have brotli module in place. I don´t know how this happend, maybe the content-encoding header is stored in the pagespeed cache with the gzip value and this overwrite the nginx-brotli encoding, but this happen, the optimized resource is served gziped not brotli encoded.

[jmarantz]

I think that makes sense. PageSpeed doesn't know anything about Brotli right now so if it is configured to cache gzipped content and send it to clients, the Brotli module may not want to unzip and then brotli-compress the resources on the fly.

So you need to disable the PageSpeed feature to cache gzipped content to enable the Brotli module to do it's thing.

It would be much better if PageSpeed could actually store Brotli-compressed assets in its cache. We had a design for this quite sometime back, but this never got implemented.