brotli compression support

[hsntgm]

trafficstars

Hello,

I use kjdev/apache-mod-brotli. It is working for apache unofficially now. The problem is w3 total cache is serving minified js/css files with gzip compressed. Can you support fully brotli compression in your fixed build or any temprorary suggest maybe rewrite rule solution for fully brotli compression integration with w3 total cache.

This response header for not minified js file;

accept-ranges:bytes cache-control:max-age=31536000, public content-encoding:br content-length:15591 content-type:application/x-javascript date:Mon, 20 Feb 2017 16:18:10 GMT etag:"f3b6-548ad8fef1ca6-br" expires:Tue, 20 Feb 2018 16:18:10 GMT last-modified:Thu, 16 Feb 2017 22:46:04 GMT pragma:public server:Apache/2.4.25 (Ubuntu) status:200 strict-transport-security:max-age=63072000; includeSubDomains; preload vary:Accept-Encoding x-content-type-options:nosniff x-frame-options:DENY

This response header for w3 minified js file;

accept-ranges:bytes cache-control:max-age=31479859, public, public content-encoding:gzip content-length:158068 content-type:application/x-javascript date:Mon, 20 Feb 2017 16:18:10 GMT etag:"26974-548eb89c77773" expires:Tue, 20 Feb 2018 00:42:29 GMT last-modified:Mon, 20 Feb 2017 00:42:29 GMT pragma:public server:Apache/2.4.25 (Ubuntu) status:200 strict-transport-security:max-age=63072000; includeSubDomains; preload vary:Accept-Encoding x-content-type-options:nosniff x-frame-options:DENY

Sincelery,

[wp-networks]

brotli support would be a great feature to add, gets a +1 from me for sure :)

[amiga-500]

Seems interesting! I figure adding support for kjdev's brotli php extension would be the quickest way to go. And while adding it, i figure adding support for the growing popularity of Zstandard too via kjdev's other php extesnion could be useful. It would modify Browser Cache's Enable HTTP (gzip) compression to be something like this instead:

Coding in the new compression support wouldn't be hard nor take that long. The longest part (and therefore the hardest) would be debugging since compression intertwines so much in w3tc. I might strongly consider adding it when i eventually get back from my extended hiatus.

Kimberly

[hsntgm]

I didn't understand why popular wordpress cache plugins doesn't support these compressions yet. Drupal has already built in brotli support. Maybe change time to other cms.

Kimberly, if i remove gzip compression tick in browser cache all minified files is serving with br compressed but page cache still use gzip and in my test when i use brotli, my page load time increased 3 sec to 8 sec! even my total page size decreased 1.5mb to 0.9 mb so there is something really wrong wp,w3 and brotli combination and it seems you must deeply analysis while integrate it to w3 total cache and wp.Also i complied on my server apache_mod_brotli not extension php_ext_brotli.

With gzip,

Wit br,

What i see maybe brotli take much time while compressing large sized files on server side ! And already issued here : https://github.com/google/brotli/issues/222

Thanks,

[amiga-500]

Thanks for the snapshots @hsntgm . What dark theme are you using? I like it!

Anyway, others can correct me if i'm wrong but how w3tc works when the gzip option is enabled is it uses php's zlib extension. It uses it to compress then write out to disk gzip files for things like html, css, and js files. It then feeds this compressed version back to client-side browsers directly, after detecting that the requesting browser's "Accepted-Encoding" header allows for it. All subsequent requests would then be from that previously compressed, saved-to-disk version. Notice the server is not involved with compressing.

In your case, when you deactivated gzip, Apache is configured to use your Brotli module for those non-compressed files. Depending on how your server is configured, it could be compressing these files each time a request is being made prior to serving it back to the client. This could explain why you see a page load time spike vs gzip. Recall, the gzips are being served back directly by w3tc from a disk store.

As such, w3tc would need to either support an existing Brotli php extension or code it in from scratch. Kjdev made your Apache Brotli module and seems to work very well. I figure supporting his equivalent php extension is the better way than to code from the ground up. It would therefore require the user (ie, you) to have his php extension already available in your php environment so w3tc can detect and make the Brotli compression option available to be selected. The php extension is only needed so w3tc can write out the compressed file once, to serve back to many. This would be a more efficient approach than having Apache compressing files constantly (slowing down server response times).

I agree with you that Brotli should be supported and it wouldn't take me long to add it. I would however need to spend a week or so after adding it in to debug every nook and cranny since w3tc makes use of compression in many areas. Sadly, i won't be rolling up my sleeves on this until i return later.

Cheers Kimberly :octocat:

[szepeviktor]

What dark theme are you using? I like it!

Maybe Firefox?

[hsntgm]

Yes, it is super cool ! You can get Firefox Developer Edition..

@amiga-500 @szepeviktor I can help you server-side tests for both nginx and apache. I can share my full phpinfo with you privately if you need it. But that's all I can do for you. I am interested in system administration mostly so my programming knowledge is not as good as you are.

Not all browsers support brotli so i thought i need still mod_deflate and have to be enabled in apache. Then the problem which compressed file my server will serve? Drupal resources helped me to serve correct compressed file according to client support with below .htaccess codes. Maybe a start point for you to work on it.

# Rules to correctly serve gzip compressed CSS and JS files.
# Requires both mod_rewrite and mod_headers to be enabled.
<IfModule mod_headers.c>
  # Serve brotli compressed CSS files if they exist and the client accepts brotli.
  RewriteCond %{HTTP:Accept-encoding} br
  RewriteCond %{REQUEST_FILENAME}\.br -s
  RewriteRule ^(.*)\.css $1\.css\.br [QSA]

  # Serve gzip compressed CSS files if they exist and the client accepts gzip.
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{REQUEST_FILENAME}\.gz -s
  RewriteRule ^(.*)\.css $1\.css\.gz [QSA]

  # Serve brotli compressed JS files if they exist and the client accepts brotli.
  RewriteCond %{HTTP:Accept-encoding} br
  RewriteCond %{REQUEST_FILENAME}\.br -s
  RewriteRule ^(.*)\.js $1\.js\.br [QSA]

  # Serve gzip compressed JS files if they exist and the client accepts gzip.
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{REQUEST_FILENAME}\.gz -s
  RewriteRule ^(.*)\.js $1\.js\.gz [QSA]

  # Serve correct content types, and prevent mod_deflate double gzip.
  RewriteRule \.css\.gz$ - [T=text/css,E=no-gzip:1]
  RewriteRule \.js\.gz$ - [T=text/javascript,E=no-gzip:1]
  RewriteRule \.css\.br$ - [T=text/css,E=no-gzip:1]
  RewriteRule \.js\.br$ - [T=text/javascript,E=no-gzip:1]

  <FilesMatch "(\.js\.gz|\.css\.gz)$">
    # Serve correct encoding type.
    Header set Content-Encoding gzip
    # Force proxies to cache gzipped & non-gzipped css/js files separately.
    Header append Vary Accept-Encoding
  </FilesMatch>
  <FilesMatch "(\.js\.br|\.css\.br)$">
    # Serve correct encoding type.
    Header set Content-Encoding br
    # Force proxies to cache gzipped & non-gzipped css/js files separately.
    Header append Vary Accept-Encoding
  </FilesMatch>
</IfModule>

I used BrotliCompressionLevel 10 which seems best stable compression level.

# Load module
LoadModule brotli_module modules/mod_brotli.so

<IfModule brotli_module>
  # Output filter
  AddOutputFilterByType BROTLI text/html text/plain text/css text/xml

  # SetOutputFilter BROTLI
  # SetEnvIfNoCase Request_URI \.txt$ no-br

  # Compression
  ## BrotliCompressionLevel: 0-11 (default: 11)
  BrotliCompressionLevel 10

  ## BrotliWindowSize: 10-24 (default: 22)
  BrotliWindowSize 22

  # Specifies how to change the ETag header when the response is compressed
  ## BrotliAlterEtag: AddSuffix, NoChange, Remove (default: AddSuffix)
  BrotliAlterEtag AddSuffix

  # Filter note
  BrotliFilterNote Input  brotli_in
  BrotliFilterNote Output brotli_out
  BrotliFilterNote Ratio  brotli_ratio

  LogFormat '"%r" %{brotli_out}n/%{brotli_in}n (%{brotli_ratio}n)' brotli
  CustomLog logs/access_log brotli
</IfModule>

I am keep in touch, Sincerely,

[nigrosimone]

@hsntgm this require a huge work because gzip is everywhere in w3tc and it seems a niche features.

[lukecav]

Does look like Gzip is used in quite a few places in core. https://github.com/szepeviktor/w3-total-cache-fixed/search?utf8=%E2%9C%93&q=gzip&type=

[AWOL-TECH]

Hi all,

Was there any progress made on this? I have recently started using brotli now that there is an official apache module - https://httpd.apache.org/docs/trunk/mod/mod_brotli.html - which can also be added in cpanel via Easyapache 4 with a default config (literally click the ea4 slider and provision and that's you compressing via brotli additonal config is optional)

I was wondering about a setting in the plugin that can auto detect the presence of mod_brotli in apache (maybe via loaded modules in php.ini?) and then prompt the user or just automatically disable gzip? Could be the easiest method after finding a way to replace gzip in the plugin core