incubator-pagespeed-ngx

incubator-pagespeed-ngx copied to clipboard

We have been using pagespeed with varying levels of success, overall it works great until it breaks.

The application is using two (2) nginx/pagespeed reverse proxies with a shared Redis cache in front of the main application nginx web servers.

Everything is running in Docker.

The nginx/pagespeed containers are StatefulSets each with a separate persistent storage volume holding the pagespeed file caches which persist across deployments.

What I noticed is that the utilization of redis cache keeps growing and growing until it maxes out the entire database (which was set to 1GB).

At some point I start seeing random 404s being returned on some pagespeed re-written images (whether this is related to the redis behavior or not I do not know).

The images which return a 404 are at random, no specific type or directory or anything, however the same 404s persist after multiple page reloads (meaning once an image turns to a 404 it remains a 404).

The 404 image files appear re-written (like all other resources) to something like (for example): 3Ab38202a868b1020cae856e23fc2caa191b79f5c959,qw=265,ah=265.pagespeed.ic.SJOX1Ol1WO.webp

Flushing the entire pagespeed cache does not resolve the issue. It is also worth noting that flushing the cache never seems to reduce the size of the redis cache, it just keeps growing over time.

Flushing the specific URL of the 404 images also does not seem to make any difference.

Can anyone help with this? I have tried moving from redis to memcached (same system exactly) to see if this works better, but would really like to understand what is wrong here.

This is the pagespeed nginx config:

` upstream application_server { server example-app:8090; keepalive 30; }

include inc/cloudflare.conf;

pagespeed MessageBufferSize 100000; pagespeed ProcessScriptVariables all; pagespeed CreateSharedMemoryMetadataCache "/var/www/pc/pagespeed/" 51200;

server {

set $root_dir "example";
set $root_domain "example.com";

listen 80;
listen [::]:80;
listen 443 ssl http2;
server_name _;

error_log  /dev/stderr;
access_log /dev/stdout main;
root /var/www/$root_dir/public;

ssl_certificate /etc/nginx/cert.pem;
ssl_certificate_key /etc/nginx/key.pem;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
ssl_session_cache shared:SSL:64m;
ssl_session_timeout 1h;

gzip on;
gzip_vary on;
gzip_disable "msie6";
gzip_min_length 256;
gzip_buffers 16 8k;
gzip_proxied any;
gzip_types
    text/plain
    text/css
    text/js
    text/xml
    text/javascript
    font/ttf
    font/opentype
    image/svg+xml
    application/javascript
    application/x-javascript
    application/json
    application/xml
    application/rss+xml
    application/ttf
    application/x-ttf
    application/x-font-otf
    application/font-woff
    application/font-woff2
    application/vnd.ms-fontobject;

pagespeed on;
pagespeed FetchHttps disable;
pagespeed AllowVaryOn "Auto";
pagespeed EnableCachePurge on;
pagespeed SupportNoScriptEnabled false;
pagespeed InPlaceResourceOptimization on;
pagespeed LogDir "/var/www/pc/pagespeed/logs/";

pagespeed LRUCacheKbPerProcess 8192;
pagespeed LRUCacheByteLimit 16384;

pagespeed FileCachePath "/var/www/pc/pagespeed/";
pagespeed FileCacheSizeKb 728000;
pagespeed FileCacheInodeLimit 5000000;
pagespeed FileCacheCleanIntervalMs 36000;

pagespeed MemcachedServers "10.1.240.5:11211,10.1.240.6:11211";

pagespeed MapOriginDomain http://127.0.0.1 https://$root_domain;
pagespeed MapOriginDomain http://127.0.0.1 https://stage-00.$root_domain;
pagespeed MapOriginDomain http://127.0.0.1 https://stage-01.$root_domain;

pagespeed EnableFilters rewrite_css;
pagespeed EnableFilters lazyload_images;
pagespeed EnableFilters rewrite_javascript;
pagespeed EnableFilters add_instrumentation;
pagespeed EnableFilters insert_dns_prefetch;
pagespeed EnableFilters flatten_css_imports;
pagespeed EnableFilters collapse_whitespace;
pagespeed EnableFilters insert_dns_prefetch;
pagespeed EnableFilters inline_google_font_css;
pagespeed EnableFilters hint_preload_subresources;
pagespeed EnableFilters in_place_optimize_for_browser;
pagespeed EnableFilters canonicalize_javascript_libraries;

pagespeed EnableFilters rewrite_images;
pagespeed EnableFilters recompress_images;
pagespeed EnableFilters convert_jpeg_to_webp;
pagespeed EnableFilters convert_to_webp_animated;

pagespeed DisableFilters responsive_images;
pagespeed DisableFilters prioritize_critical_css;

pagespeed Disallow "*/admin*";
pagespeed Disallow "*/healthz";
pagespeed Disallow "*/ping.js";
pagespeed Disallow "*/ping.php";
pagespeed Disallow "*/pixel.gif";
pagespeed Disallow "*/blank.gif";

pagespeed Statistics on;
pagespeed StatisticsLogging on;

pagespeed StatisticsPath /ngx_pagespeed_statistics;
pagespeed MessagesPath /ngx_pagespeed_message;
pagespeed ConsolePath /pagespeed_console;
pagespeed AdminPath /pagespeed_admin;

## Ensure requests for pagespeed optimized resources go to the pagespeed handler
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
    add_header "" "";
}

location ~ "^/pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }

## PageSpeed Admin UI paths
location /ngx_pagespeed_statistics {
    include inc/restrictions.conf;
}
location /ngx_pagespeed_global_statistics {
    include inc/restrictions.conf;
}
location /ngx_pagespeed_message {
    include inc/restrictions.conf;
}
location /pagespeed_console {
    include inc/restrictions.conf;
}
location ^~ /(pagespeed_global_admin|pagespeed_admin) {
    include inc/restrictions.conf;
}

location / {
    proxy_pass http://application_server;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header Connection "Keep-Alive";
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Proto $scheme;
}

include inc/redirects.conf;

} `

Hi At a first glance you have a https site, I guess that redirects.conf file make the rigth redirect http -> https. I don´t think that the 404 error is caused by redis. The cause of these error type is that pagespeed is unable to fecht the url. What happens when you add /?PageSpeedFilters=+debug to the main url? This parameter put debug messages in the html code.

Hi, first of all thank you for your reply :)

The redirects.inc are just for some old urls to 301 them to new pages, they don't do anything with pagespeed.

HTTPS terminates at the pagespeed reverse proxy, upstream the application webserver communicates over HTTP which is why I have: proxy_pass http://application_server;

Along with: pagespeed MapOriginDomain http://127.0.0.1 https://$root_domain;

I'm not sure about the +debug, I've disabled image re-writing on production because of this and cannot just re-enable it to test. What i can tell you though is that for a few weeks all was fine, then new images were added to the site and they got rendered as 404s.

At the same time I noticed that the redis DB was 99.9999% full (potentially un-realted, although there was 1GB of space in it to be used).

We store images in a separate redis DB not on file and they are served via php. The separate redis db is in a cluster too.

What would happen if:

#1 webadmin uploads image to master redis #2 pagespeed from application instance requests it from redis slave which is not yet up to date #3 pagespeed hits a 404 on first attempt #4 data propagates the redis slaves #5 pagespeed tries again? Does it find it or persists with 404?

BTW, the images are also behind an nginx cache, so on first request the application nginx get an image via PHP, but then pulls it from its cache (since rendering every time is very wasteful).

However having said all this, only the newly added images broke old ones were fine, then i flushed the cached and a few older ones broke too so I removed the rewrite_images.

It's strange.

The redirects.inc are just for some old urls to 301 them to new pages, they don't do anything with pagespeed. I´m not talkin about pagespeed but about you site. In the config you have posted the server is listening in ports 80 and 443 then w/o any redirect from port 80 to the 443 you have 2 identical sites, 1 for http and 1 for https. Google can penalize with duplicate contents. I'm not sure about the +debug, I've disabled image re-writing on production because of this and cannot just re-enable it to test. What i can tell you though is that for a few weeks all was fine, then new images were added to the site and they got rendered as 404s.

The +debug is a url parameter you can set in the url: https://yourdomain.com/?PageSpeedFilters=+debug This paremeter inject debug messages in the html code, you can view it viewing the source code from the page.

At the same time I noticed that the redis DB was 99.9999% full (potentially un-realted, although there was 1GB of space in it to be used).

Maybe the problem is the redis db size. Can you try to increase it? The pagespeed cache need to be larger than all the resources optimized cause it need to store several image versions for example. The redis at 99% is a spected behaviour cause pagespeed don´t have a clean process for it and then the clean process is the allkeys-lru from redis.

If pagespeed can´t optimize a resource then it remember this fail for a time, I don´t remember if this time is a fixed amount or configurable. You can see these messages in the pagespeed log: remembering recent failure for 299 seconds

OK, so what i have done till now is re-build nginx with the cache purge module and set up the communication between it and the pagespeed nginx.

Changed the Forbid filters to Disallow, so i can enable them via the URL along with the debug option.

Moved away from redis (says it's a bit experimental on the pagespeed site), and changed to a 2 node memcached setup, of 2GB i think. Should be more than enough.

I'll test it over the weekend or next week and update this thread, hopefully it works and helps others out :)

I'll keep an eye out for purge requests being passed downstream and upstream to the memcached instances.

Redis is in a stable state. And have the advantage to don´t limit the object size. With memchached by default you have a 2Mb objec size limit, can be changed with a start parameter, but still a fixed max size. With redis you don´t have this limit

Hi, So you think redis is a better option than Memcached? I should not bother with Memcached?

2MB should be more than enough per object, that does not worry me, but regardless you suggest redis over Memcached?

I just want 100% stability, if there is no more space in the cache I would like the image to just be left as is rather than a 404, cause this causes quite a bit of issues with my colleagues, then again we have people uploading 200Kb thumbnails and pagespeed gets them down to 20Kb so that is worth having.

So, i checked the debug message on the 404 with redis as the cache and this is what it says:

The image was not inlined because it has too many bytes. Image https://example.com/file/media:f71200145e879c374114ccb9ea62e2c01c7920e342 does not appear to need resizing.

Then it serves this: /file/xmedia,3Af71200145e879c374114ccb9ea62e2c01c7920e342.pagespeed.ic.NeQ8AuMyvH.webp

Ok, so I have an interesting update @Lofesa .

I pushed the new setup to production and got 404s on pagespeed images immediatly.

Let me explain the setup of the pagespeed nginx servers.

There are two of them and each is running as a docker container. Each pagespeed instance is set to write its file cache to a mounted hard drive (there are 2 separate hard drives) one per pagespeed server.

The data on this hard drive persists across re-deployments of the pagespeed nginx servers, which happens every time the codebase is deployed.

It got me thinking, to see 404s on a new empty memcached DB means that the DB being full has nothing to do with it, but it is likely tied to the local file metatdata cahe or whatever.

What am i doing wrong? Should I not persist this data and let it expire on each deployment? This is the file cache:

pagespeed FileCachePath "/var/www/pc/pagespeed/";
pagespeed FileCacheSizeKb 728000;
pagespeed FileCacheInodeLimit 5000000;
pagespeed FileCacheCleanIntervalMs 36000;

/pc is the root of the mounted persistent drive.

There is also this:

pagespeed LRUCacheKbPerProcess 8192;
pagespeed LRUCacheByteLimit 16384;

Should i get rid of this file cache? Is it possible to set up pagespeed so that 100% (everything) is in memcached or redis?

Alternatively, what if i disabled this: https://www.modpagespeed.com/doc/filter-cache-extend

It would leave the image URLs as they were right? and like that any missing image would just get the default file instead?

It does not matter if the setup is not 100% efficient in terms or optimisation.

Thanks Andrew

Further update, seems that adding these does nothing to prevent PageSpeed from re-writing the urls

pagespeed DisableFilters extend_cache;
pagespeed DisableFilters extend_cache_images;

In parts, as Jack the ripper would say =:-))

Yes, I think redis is better that memcached for the pagespeed purposes. They have no object size limit and can be scaled configuring a cluster.

The info you get from the +debug is not relevant to the issue. You have enabled the inline image filter and have a max size the image can have to be inlined (if tou don´t have this max size, this parameter have a default value)

The fact that you have set 2 different file-cache is the most relevant info. When you have 2 servers spreading the same files, you need that the cache are the SAME for bath servers. This can be achieved by 2 ways: 1.- A shared file system. This is advised againts it, shared filesystems add to much latency 2.- A shared database, like memcached or redis.

Yuou need to configure pagespeed FileCachePath in both servers, but you need to be sure nothing is writen in thses paths. And again, if a object does not fit in the memcache db, then this object is written to the disk. What could be happening here? Well, say you make a request to the server A, this make all the html come from server A to the client, but before this, pagespeed does it magic and rewrite the resources in the server A pagespeed cache When the client have the html, it does request for the resources, if these resources are in server A pagespeed cache and the request go to that server all is ok, but if the request goes to server B it know nothing about this rewrited resource.

Hi @Lofesa - yes you are right, in fact if i refresh the page a few times, sometimes the missing images show and other times they do not, depends on which server is being hit.

So, the issue now is that a shared file system is not possible on Google hosting with docker unfortunately.

Both servers point to the same Memcached instance for the cache, however they both have a FileCachePath. I don't know how to make sure that PageSpeed does not use anything except the Memcached DB, is this even possible?

In an ideal world it would be n instances working with one set of caches and metadata, but I don't know how to achieve this with PageSpeed, is it possible?

Should I disable the file cache and leave only Memcached? Can this be done?

The documentation says 'A file-system based cache is always employed on each server.' https://www.modpagespeed.com/doc/system#server_cache

So I dunno what to do to be honest to keep both instances in sync.

There is also this in the config: pagespeed CreateSharedMemoryMetadataCache "/var/www/pc/pagespeed/" 102400;

However that directory is not shared between both servers, is that a bad setup?

Read this thread https://github.com/apache/incubator-pagespeed-ngx/issues/1679 Is not about FileCache only, but with DefaultSharedMemoryCache. FileCachePath MUST be configured, even if not used. If no FileCachePath is set, then pagespeed refuses to start

Thanks @Lofesa

From that thread what i took away was to set: pagespeed DefaultSharedMemoryCacheKB 0;

And to set (we have a few different domains the website is served on but they are all the same site): pagespeed CacheFragment $root_dir;

Other than that this is the only thing which worries me (site has been unchanged for last day, no new content), memcached data usage: https://ibb.co/6wPNqw2

Hi -- I want to clarify some confusion on this thread.

When you have multiple servers running mod_pagespeed, they do not need to share a cache(*). Thinking about this another way, caches generally may evict assets at any time, and the system needs to work correctly. So the URL generated by mod_pagespeed has all the information that is needed to reconstruct the optimized asset from the original one.

If 404s are being served, it is *not *because the caches are separate. It is most likely because of a bug, or because one of the instances is unable to fetch the origin resource for some reason.

It is beneficial to use a shared cache setup like memcached or redis, because it improves performance (response time, and likelihood of getting an optimized asset).

• exceptions to the above:

  1. the outline_javascript and outline_styles filters depend on caches not evicting. We don't recommend their use :)
  2. If you use LoadFromFile then you definitely need access to the file-system where the assets are from each resource.

More detail on how caching works in mod_pagespeed: http://modpagespeed.jmarantz.com/2012/12/caching-in-modpagespeed.html

Hope this helps, -Josh

On Sun, May 3, 2020 at 7:58 AM Andrew Borg [email protected] wrote:

Thanks @Lofesa https://github.com/Lofesa

From that thread what i took away was to set:

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pagespeed-ngx/issues/1686#issuecomment-623098797, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAO2IPK4I3B4UFISG7REJH3RPVL6NANCNFSM4MTB4TGA .

I have a sneaking suspicion that it might have been my redis config to skrew it all up.

The redis config was set with allkeys-lru - https://redis.io/topics/lru-cache

Thinking about it, could it be that as the redis DB became full, it evicted some data which PageSpeed thought still existed in the DB, thus rendering links on images with 404?

@jmarantz Not using either outline_javascript or outline_styles. I'd like to have a single cache because it seems so much more efficient to do so.

Anyways, not sure if it's the case that the allkeys-lru caused the issue or not, however there is the fact that the issue only came up when the DB was maxed out.

Further information, we have two sites which used to share the same redis cache and on the other site there were 404s too, i noticed them today and moved that site onto the new setup with memcached.

Only time will tell if this is the case or not, but it could take months before it re-surfaces, naturally.

I don't think that should cause 404s. It might cause unoptimized content to be served while the cache is warned up.

On Sun, May 3, 2020, 2:32 PM Andrew Borg [email protected] wrote:

I have a sneaking suspicion that it might have been my redis config to skrew it all up.

The redis config was set with allkeys-lru - https://redis.io/topics/lru-cache

Thinking about it, could it be that as the redis DB became full, it evicted some data which PageSpeed thought still existed in the DB, thus rendering links on images with 404?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pagespeed-ngx/issues/1686#issuecomment-623158601, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAO2IPJVMP4RXVF5ESQL3HTRPW2FBANCNFSM4MTB4TGA .

Then i'm at a loss, dunno what else to look at. @jmarantz

BTW, is this normal, every so often the cache usage just jumps up for no reason: https://ibb.co/S0L17hQ

Sorry, didn't mean to say it was a dead end. For an asset that's 404ing in the browser, can you see what's happening on the Apache server logs? Maybe give more details about that. For example, if you found a resource was failing named foo.css.pagespeed.cf.HASH.css in the browser, can you scan the Apache error logs to see if there's issues fetching foo.css?

In a multi-server setup, it could be that one of the servers can fetch foo.css, and the other one can't. A shared redis might hide this problem and make it reproduce inconsistently, but ultimately the root cause is likely a failure to fetch the resource from mod_pagespeed, and that should show up in the logs.

However, another scenario is that you are running into this bug: https://github.com/apache/incubator-pagespeed-mod/issues/1145. It took me a while to find that. I haven't fully re-read it, but I think the scenario could be that neither of your pagespeed instances is able to fetch anything. And the way it is able to optimize assets is by saving them in the local cache while it serves them from their original URLs via In Place Resource Optimization https://www.modpagespeed.com/doc/system#ipro.

To test this theory, you could disable In Place Resource Optimization off:

ModPagespeedInPlaceResourceOptimization off

If you do this, I suspect the 404s will go away, but you also won't be optimizing anything. Then we can work on how to ensure your mod_pagespeed instances are able to fetch the origin assets via HTTP. Can you give that first step a try? Disable the feature as above, then see if that fixes the 404s.

-Josh

On Sun, May 3, 2020 at 3:28 PM Andrew Borg [email protected] wrote:

Then i'm at a loss, dunno what else to look at.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pagespeed-ngx/issues/1686#issuecomment-623167179, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAO2IPIWX6E3FR7TNXPYBRTRPXAWRANCNFSM4MTB4TGA .

Hi @jmarantz I think in this scenario:

• 2 servers (A,B) w/ pagespeed that don´t share the metadata cache, both are proxy-cache to a backgroud server.
• Html code is procesed by the server A, so the assets get rewrited and metadata are stored in their metadata cahe.
• A rewrited asset is requested to server B that don´t have metadata for this asset.
• What would be the output?

However, another scenario is that you are running into this bug: apache/incubator-pagespeed-mod#1145. It took me a while to find that. I haven't fully re-read it, but I think the scenario could be that neither of your pagespeed instances is able to fetch anything. And the way it is able to optimize assets is by saving them in the local cache while it serves them from their original URLs via In Place Resource Optimization https://www.modpagespeed.com/doc/system#ipro.

@jmarantz I don't think this can be the case because I see the shared cache being filled up so something is being saved to it and also the URLs of the assets are being changed to PageSpeed URLs.

So far what do we know for a fact:

1. Issue happens when Redis Shared cache is 99.9999% full
2. The Redis Shared Cache was configured with maxmemory -> allkeys-lru
3. Two websites each with two PageSpeed reverse proxies use the same Redis (although different DB numbers per website)
4. Some images which were re-written by PageSpeed have the new PageSpeed URL give a 404 but the original non-optimised image url works
5. All Images are saved in a separate Redis DB and served via PHP
6. The entire setup is using docker
7. extend_cache_images is disabled

@jmarantz @Lofesa

I have an update for you, been running pagespeed again for few days now with zero issues, so far. From the advice in this thread I've set pagespeed DefaultSharedMemoryCacheKB 0;

and this (we have many domains pointing to the same website) pagespeed CacheFragment $root_dir;

Just one thing please, take a look at this image: https://ibb.co/1T1KpdX

It's a usage graph of the cache memcache). Is this possibly normal?? It just keeps growing and we do have maybe a few thousand images on the site but nothing to fill 1.5GB of optimized content. It seems like it will not stop, just keep growing at a steady rate, how?

This makes sense and everything you've mentioned is consistent with my hypothesis for the root cause.

If your cache doesn't get into a state where it evicts anything, then it will work to: a) not be able to fetch your CSS files via http b) be able to capture them into the shared cache as they are served optimized, via an output-filter that is installed by mod_pagespeed as part of InPlaceResourceOptimization c) optimize those cached resources d) re-optimize them also in another server with a separate metadata cache, having loaded them from the shared http cache e) save the optimized resources from the shared cache. f) serve them happily from all servers connected to that shared cache.

That all works great until the asset is evicted. At that point, pagespeed gets a request for an optimized URL. It can't find it in the cache. Then it: g) decodes the URL to find the origin asset h) attempts to use HTTP to fetch the origin asset, and fails i) serves a 404.

If you are OK keeping your cache from ever evicting anything, then this bug can be tolerated. But as you observed, new resources are added to the cache all the time, and depending on the eviction algorithm, you may find your valuable (and non-reconstructable) assets evicted and 404s will result.

One theory as to why new stuff is added to the cache is that PageSpeed attempts to use the cache to store properties of an HTML page, such as which domains are referenced, or which images are above the fold and hence would benefit from inlining and suffer from lazyloading. HTML pages may have URLs with a high-entropy query-parameter, and never be revisited. That can cause a lot of cache writes, and ultimately (depending on cache eviction setup), cause you to drop critical resources that you can't reconstruct.

This is what https://github.com/apache/incubator-pagespeed-mod/issues/1145 is all about.

The best solution to this I think is to fix HTTP fetching so it works in your environment. Can you check to see whether it is, you can look at the pagespeed_admin/statistics page. For example, the stats for modpagesped.com look like this:

[image: Screen Shot 2020-05-05 at 4.55.44 PM.png]

The name of the HTTP fetcher used by mod_pagespeed is "serf", and this page tells us that modpagespeed.com's installation of mod_pagespeed has done 5425 successful fetches since the server started, and zero fetches failed. What do the stats look like on your system?

On Tue, May 5, 2020 at 4:25 PM Andrew Borg [email protected] wrote:

@jmarantz https://github.com/jmarantz @Lofesa https://github.com/Lofesa

I have an update for you, been running pagespeed again for few days now with zero issues, so far. From the advice in this thread I've set pagespeed DefaultSharedMemoryCacheKB 0;

and this (we have many domains pointing to the same website) pagespeed CacheFragment $root_dir;

Just one thing please, take a look at this image: https://ibb.co/1T1KpdX

It's a usage graph of the cache memcache). Is this possibly normal?? It just keeps growing and we do have maybe a few thousand images on the site but nothing to fill 1.5GB of optimized content. How?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pagespeed-ngx/issues/1686#issuecomment-624286853, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAO2IPO2PVMCXRQCMPYJACLRQBY2JANCNFSM4MTB4TGA .

I cannot see the image you posted so i'm not sure exactly what you are looking for, but these are the all the statistics below. I'll look into checking whether the images are accessible tomorrow, i'll try disable In Place Optimisation and see that the assets are reachable.

active_expensive_operations | 0 | 0 cache_backend_hits | 3561103 | 1 cache_backend_misses | 653875 | 1 cache_batcher_coalesced_gets | 0 | 0 cache_batcher_dropped_gets | 0 | 0 cache_batcher_queued_gets | 0 | 0 cache_deletes | 0 | 0 cache_expirations | 457433 | 1 cache_extensions | 0 | 0 cache_fallbacks | 36750 | 1 cache_flush_count | 2 | 1 cache_flush_timestamp_ms | 0 | 0 cache_hits | 758962 | 1 cache_inserts | 76820 | 1 cache_misses | 3456016 | 1 cache_time_us | 1418254904 | 1 central_controller_reconnect_time_ms | 0 | 0 child_shutdown_count | 0 | 0 compressed_cache_compressed_size | 232267172 | 1 compressed_cache_corrupt_payloads | 0 | 0 compressed_cache_original_size | 1428624411 | 1 converted_meta_tags | 259130 | 1 critical_css_beacon_filter_script_added_count | 0 | 0 critical_css_no_beacon_due_to_missing_data | 0 | 0 critical_css_skipped_due_to_charset | 0 | 0 critical_images_beacon_filter_script_added_count | 0 | 0 critical_images_expired_count | 0 | 0 critical_images_not_found_count | 0 | 0 critical_images_valid_count | 0 | 0 css_combine_opportunities | 8857 | 1 css_elements_moved | 0 | 0 css_file_count_reduction | 0 | 0 css_filter_blocks_rewritten | 708 | 1 css_filter_fallback_failures | 0 | 0 css_filter_fallback_rewrites | 0 | 0 css_filter_parse_failures | 0 | 0 css_filter_rewrites_dropped | 0 | 0 css_filter_total_bytes_saved | 94586 | 1 css_filter_total_original_bytes | 113450969 | 1 css_filter_uses | 450867 | 1 css_imports_to_links | 0 | 0 current_expensive_operations | 0 | 0 current_fetch_queue_size | 0 | 0 domain_rewrites | 0 | 0 downstream_cache_purge_attempts | 3838 | 1 file_cache_bytes_freed_in_cleanup | 0 | 0 file_cache_cleanups | 0 | 0 file_cache_deletes | 0 | 0 file_cache_disk_checks | 0 | 0 file_cache_evictions | 0 | 0 file_cache_hits | 0 | 0 file_cache_inserts | 0 | 0 file_cache_misses | 0 | 0 file_cache_skipped_cleanups | 0 | 0 file_cache_started_cleanups | 0 | 0 file_cache_write_errors | 0 | 0 flatten_imports_charset_mismatch | 0 | 0 flatten_imports_complex_queries | 0 | 0 flatten_imports_invalid_url | 0 | 0 flatten_imports_limit_exceeded | 0 | 0 flatten_imports_minify_failed | 0 | 0 flatten_imports_recursion | 0 | 0 font_service_input_resource_hit | 0 | 0 font_service_input_resource_miss | 0 | 0 font_service_input_resource_recent_fetch_failure | 0 | 0 font_service_input_resource_recent_uncacheable_failure | 0 | 0 font_service_input_resource_recent_uncacheable_miss | 0 | 0 google_analytics_page_load_count | 0 | 0 google_analytics_rewritten_count | 0 | 0 html_worker_queue_depth | 0 | 0 http_approx_header_bytes_fetched | 9858012 | 1 http_bytes_fetched | 605029414 | 1 http_fetches | 27058 | 1 image_file_count_reduction | 0 | 0 image_inline | 9052922 | 1 image_norewrites_high_resolution | 88 | 1 image_ongoing_rewrites | 0 | 0 image_resized_using_rendered_dimensions | 0 | 0 image_rewrite_latency_total_ms | 1079050 | 1 image_rewrite_total_bytes_saved | 692188535 | 1 image_rewrite_total_original_bytes | 1098453869 | 1 image_rewrite_uses | 14890738 | 1 image_rewrites | 15342 | 1 image_rewrites_dropped_decode_failure | 0 | 0 image_rewrites_dropped_intentionally | 2981 | 1 image_rewrites_dropped_mime_type_unknown | 1616 | 1 image_rewrites_dropped_nosaving_noresize | 821 | 1 image_rewrites_dropped_nosaving_resize | 456 | 1 image_rewrites_dropped_server_write_fail | 0 | 0 image_webp_alpha_timeouts | 0 | 0 image_webp_conversion_gif_animated_timeouts | 0 | 0 image_webp_conversion_gif_timeouts | 0 | 0 image_webp_conversion_jpeg_timeouts | 0 | 0 image_webp_conversion_png_timeouts | 0 | 0 image_webp_opaque_timeouts | 0 | 0 image_webp_rewrites | 5370 | 1 in_place_oversized_opt_stream | 0 | 0 in_place_uncacheable_rewrites | 0 | 0 inserted_ga_snippets | 0 | 0 instrumentation_filter_script_added_count | 272556 | 1 ipro_not_in_cache | 1619291 | 1 ipro_not_rewritable | 2382024 | 1 ipro_recorder_dropped_due_to_load | 122166 | 1 ipro_recorder_dropped_due_to_size | 1 | 1 ipro_recorder_failed | 1616819 | 1 ipro_recorder_inserted_into_cache | 2468 | 1 ipro_recorder_not_cacheable | 33586 | 1 ipro_recorder_resources | 1619291 | 1 ipro_served | 159357 | 1 javascript_blocks_minified | 1001687 | 1 javascript_did_not_shrink | 386554 | 1 javascript_failed_to_write | 0 | 0 javascript_libraries_identified | 0 | 0 javascript_minification_disabled | 0 | 0 javascript_minification_failures | 182060 | 1 javascript_minify_uses | 797346 | 1 javascript_reducing_minifications | 797183 | 1 javascript_total_bytes_saved | 601286501 | 1 javascript_total_original_bytes | 2305755303 | 1 js_file_count_reduction | 0 | 0 low_priority_worked_queue_depth | 0 | 0 lru_cache_deletes | 0 | 0 lru_cache_hits | 0 | 0 lru_cache_inserts | 0 | 0 lru_cache_misses | 0 | 0 memcache_error_burst_size | 0 | 0 memcache_last_error_checkpoint_ms | 0 | 0 memcache_timeouts | 0 | 0 memcached_async_deletes | 0 | 0 memcached_async_hits | 0 | 0 memcached_async_inserts | 0 | 0 memcached_async_misses | 0 | 0 memcached_blocking_deletes | 0 | 0 memcached_blocking_hits | 0 | 0 memcached_blocking_inserts | 0 | 0 memcached_blocking_misses | 0 | 0 named_lock_rewrite_scheduler_locks_held | 0 | 0 not_cacheable | 0 | 0 num_cache_control_not_rewritable_resources | 20586 | 1 num_cache_control_rewritable_resources | 34060 | 1 num_conditional_refreshes | 2131 | 1 num_css_inlined | 0 | 0 num_css_not_used_for_critical_css_computation | 0 | 0 num_css_used_for_critical_css_computation | 0 | 0 num_deadline_alarm_invocations | 10 | 1 num_dedup_inlined_images_candidates_found | 0 | 0 num_dedup_inlined_images_candidates_replaced | 0 | 0 num_fallback_responses_served | 0 | 0 num_fallback_responses_served_while_revalidate | 0 | 0 num_flushes | 831325 | 1 num_js_inlined | 0 | 0 num_local_storage_cache_candidates_added | 0 | 0 num_local_storage_cache_candidates_found | 0 | 0 num_local_storage_cache_candidates_removed | 0 | 0 num_local_storage_cache_stored_css | 0 | 0 num_local_storage_cache_stored_images | 0 | 0 num_local_storage_cache_stored_total | 0 | 0 num_proactively_freshen_user_facing_request | 0 | 0 num_resource_fetch_failures | 15876 | 1 num_resource_fetch_successes | 6478 | 1 num_rewrites_abandoned_for_lock_contention | 2519 | 1 page_load_count | 111093 | 1 pcache_cohorts_beacon_cohort_deletes | 0 | 0 pcache_cohorts_beacon_cohort_hits | 643846 | 1 pcache_cohorts_beacon_cohort_inserts | 92601 | 1 pcache_cohorts_beacon_cohort_misses | 172069 | 1 pcache_cohorts_dependencies_cohort_deletes | 0 | 0 pcache_cohorts_dependencies_cohort_hits | 257790 | 1 pcache_cohorts_dependencies_cohort_inserts | 169473 | 1 pcache_cohorts_dependencies_cohort_misses | 558035 | 1 pcache_cohorts_dom_deletes | 0 | 0 pcache_cohorts_dom_hits | 750667 | 1 pcache_cohorts_dom_inserts | 760324 | 1 pcache_cohorts_dom_misses | 65158 | 1 popularity_contest_num_rewrites_awaiting_retry | 0 | 0 popularity_contest_num_rewrites_running | 0 | 0 popularity_contest_queue_size | 0 | 0 purge_cancellations | 0 | 0 purge_contentions | 0 | 0 purge_file_parse_failures | 0 | 0 purge_file_stats | 0 | 0 purge_file_write_failures | 0 | 0 purge_file_writes | 0 | 0 purge_index | 0 | 0 purge_poll_timestamp_ms | 0 | 0 queued_expensive_operations | 0 | 0 redis_async_deletes | 0 | 0 redis_async_hits | 0 | 0 redis_async_inserts | 0 | 0 redis_async_misses | 0 | 0 redis_blocking_deletes | 0 | 0 redis_blocking_hits | 0 | 0 redis_blocking_inserts | 0 | 0 redis_blocking_misses | 0 | 0 redis_cluster_redirections | 0 | 0 redis_cluster_slots_fetches | 0 | 0 resource_404_count | 3367 | 1 resource_fetch_construct_failures | 26 | 1 resource_fetch_construct_successes | 58 | 1 resource_fetches_cached | 45634 | 1 resource_url_domain_acceptances | 47324330 | 1 resource_url_domain_rejections | 854756 | 1 rewrite_cached_output_hits | 42741964 | 1 rewrite_cached_output_missed_deadline | 40747 | 1 rewrite_cached_output_misses | 8206401 | 1 rewrite_worker_queue_depth | 0 | 0 serf_fetch_active_count | 0 | 0 serf_fetch_bytes_count | 0 | 0 serf_fetch_cancel_count | 0 | 0 serf_fetch_cert_errors | 0 | 0 serf_fetch_failure_count | 0 | 0 serf_fetch_last_check_timestamp_ms | 0 | 0 serf_fetch_request_count | 0 | 0 serf_fetch_time_duration_ms | 0 | 0 serf_fetch_timeout_count | 0 | 0 serf_fetch_ultimate_failure | 0 | 0 serf_fetch_ultimate_success | 0 | 0 shm_cache_deletes | 0 | 0 shm_cache_hits | 0 | 0 shm_cache_inserts | 0 | 0 shm_cache_misses | 0 | 0 show_ads_api_replaced_for_async | 0 | 0 show_ads_snippets_converted | 0 | 0 show_ads_snippets_not_converte | 0 | 0 slurp_404_count | 0 | 0 statistics_404_count | 0 | 0 stdio_fs_outstanding_ops | 0 | 0 stdio_fs_slow_ops | 0 | 0 stdio_fs_total_ops | 0 | 0 successful_downstream_cache_purges | 0 | 0 total_page_load_ms | 522503305 | 1 url_input_resource_hit | 19275 | 1 url_input_resource_miss | 22354 | 1 url_input_resource_recent_fetch_failure | 121 | 1 url_input_resource_recent_uncacheable_failure | 4589 | 1 url_input_resource_recent_uncacheable_miss | 0 | 0 url_trim_saved_bytes | 0 | 0 url_trims | 0 | 0

You are using nginx I assume as you are posting to the nginx group. Is that right?

Do you know if you are using the nginx fetcher or serf? What does your config look like? It looks like you have 0 attempts at fetching with serf, which makes me think this is a config issue.

On Tue, May 5, 2020 at 5:47 PM Andrew Borg [email protected] wrote:

I cannot see the image you posted, these are the stats:

active_expensive_operations | 0 | 0 cache_backend_hits | 3561103 | 1 cache_backend_misses | 653875 | 1 cache_batcher_coalesced_gets | 0 | 0 cache_batcher_dropped_gets | 0 | 0 cache_batcher_queued_gets | 0 | 0 cache_deletes | 0 | 0 cache_expirations | 457433 | 1 cache_extensions | 0 | 0 cache_fallbacks | 36750 | 1 cache_flush_count | 2 | 1 cache_flush_timestamp_ms | 0 | 0 cache_hits | 758962 | 1 cache_inserts | 76820 | 1 cache_misses | 3456016 | 1 cache_time_us | 1418254904 | 1 central_controller_reconnect_time_ms | 0 | 0 child_shutdown_count | 0 | 0 compressed_cache_compressed_size | 232267172 | 1 compressed_cache_corrupt_payloads | 0 | 0 compressed_cache_original_size | 1428624411 | 1 converted_meta_tags | 259130 | 1 critical_css_beacon_filter_script_added_count | 0 | 0 critical_css_no_beacon_due_to_missing_data | 0 | 0 critical_css_skipped_due_to_charset | 0 | 0 critical_images_beacon_filter_script_added_count | 0 | 0 critical_images_expired_count | 0 | 0 critical_images_not_found_count | 0 | 0 critical_images_valid_count | 0 | 0 css_combine_opportunities | 8857 | 1 css_elements_moved | 0 | 0 css_file_count_reduction | 0 | 0 css_filter_blocks_rewritten | 708 | 1 css_filter_fallback_failures | 0 | 0 css_filter_fallback_rewrites | 0 | 0 css_filter_parse_failures | 0 | 0 css_filter_rewrites_dropped | 0 | 0 css_filter_total_bytes_saved | 94586 | 1 css_filter_total_original_bytes | 113450969 | 1 css_filter_uses | 450867 | 1 css_imports_to_links | 0 | 0 current_expensive_operations | 0 | 0 current_fetch_queue_size | 0 | 0 domain_rewrites | 0 | 0 downstream_cache_purge_attempts | 3838 | 1 file_cache_bytes_freed_in_cleanup | 0 | 0 file_cache_cleanups | 0 | 0 file_cache_deletes | 0 | 0 file_cache_disk_checks | 0 | 0 file_cache_evictions | 0 | 0 file_cache_hits | 0 | 0 file_cache_inserts | 0 | 0 file_cache_misses | 0 | 0 file_cache_skipped_cleanups | 0 | 0 file_cache_started_cleanups | 0 | 0 file_cache_write_errors | 0 | 0 flatten_imports_charset_mismatch | 0 | 0 flatten_imports_complex_queries | 0 | 0 flatten_imports_invalid_url | 0 | 0 flatten_imports_limit_exceeded | 0 | 0 flatten_imports_minify_failed | 0 | 0 flatten_imports_recursion | 0 | 0 font_service_input_resource_hit | 0 | 0 font_service_input_resource_miss | 0 | 0 font_service_input_resource_recent_fetch_failure | 0 | 0 font_service_input_resource_recent_uncacheable_failure | 0 | 0 font_service_input_resource_recent_uncacheable_miss | 0 | 0 google_analytics_page_load_count | 0 | 0 google_analytics_rewritten_count | 0 | 0 html_worker_queue_depth | 0 | 0 http_approx_header_bytes_fetched | 9858012 | 1 http_bytes_fetched | 605029414 | 1 http_fetches | 27058 | 1 image_file_count_reduction | 0 | 0 image_inline | 9052922 | 1 image_norewrites_high_resolution | 88 | 1 image_ongoing_rewrites | 0 | 0 image_resized_using_rendered_dimensions | 0 | 0 image_rewrite_latency_total_ms | 1079050 | 1 image_rewrite_total_bytes_saved | 692188535 | 1 image_rewrite_total_original_bytes | 1098453869 | 1 image_rewrite_uses | 14890738 | 1 image_rewrites | 15342 | 1 image_rewrites_dropped_decode_failure | 0 | 0 image_rewrites_dropped_intentionally | 2981 | 1 image_rewrites_dropped_mime_type_unknown | 1616 | 1 image_rewrites_dropped_nosaving_noresize | 821 | 1 image_rewrites_dropped_nosaving_resize | 456 | 1 image_rewrites_dropped_server_write_fail | 0 | 0 image_webp_alpha_timeouts | 0 | 0 image_webp_conversion_gif_animated_timeouts | 0 | 0 image_webp_conversion_gif_timeouts | 0 | 0 image_webp_conversion_jpeg_timeouts | 0 | 0 image_webp_conversion_png_timeouts | 0 | 0 image_webp_opaque_timeouts | 0 | 0 image_webp_rewrites | 5370 | 1 in_place_oversized_opt_stream | 0 | 0 in_place_uncacheable_rewrites | 0 | 0 inserted_ga_snippets | 0 | 0 instrumentation_filter_script_added_count | 272556 | 1 ipro_not_in_cache | 1619291 | 1 ipro_not_rewritable | 2382024 | 1 ipro_recorder_dropped_due_to_load | 122166 | 1 ipro_recorder_dropped_due_to_size | 1 | 1 ipro_recorder_failed | 1616819 | 1 ipro_recorder_inserted_into_cache | 2468 | 1 ipro_recorder_not_cacheable | 33586 | 1 ipro_recorder_resources | 1619291 | 1 ipro_served | 159357 | 1 javascript_blocks_minified | 1001687 | 1 javascript_did_not_shrink | 386554 | 1 javascript_failed_to_write | 0 | 0 javascript_libraries_identified | 0 | 0 javascript_minification_disabled | 0 | 0 javascript_minification_failures | 182060 | 1 javascript_minify_uses | 797346 | 1 javascript_reducing_minifications | 797183 | 1 javascript_total_bytes_saved | 601286501 | 1 javascript_total_original_bytes | 2305755303 | 1 js_file_count_reduction | 0 | 0 low_priority_worked_queue_depth | 0 | 0 lru_cache_deletes | 0 | 0 lru_cache_hits | 0 | 0 lru_cache_inserts | 0 | 0 lru_cache_misses | 0 | 0 memcache_error_burst_size | 0 | 0 memcache_last_error_checkpoint_ms | 0 | 0 memcache_timeouts | 0 | 0 memcached_async_deletes | 0 | 0 memcached_async_hits | 0 | 0 memcached_async_inserts | 0 | 0 memcached_async_misses | 0 | 0 memcached_blocking_deletes | 0 | 0 memcached_blocking_hits | 0 | 0 memcached_blocking_inserts | 0 | 0 memcached_blocking_misses | 0 | 0 named_lock_rewrite_scheduler_locks_held | 0 | 0 not_cacheable | 0 | 0 num_cache_control_not_rewritable_resources | 20586 | 1 num_cache_control_rewritable_resources | 34060 | 1 num_conditional_refreshes | 2131 | 1 num_css_inlined | 0 | 0 num_css_not_used_for_critical_css_computation | 0 | 0 num_css_used_for_critical_css_computation | 0 | 0 num_deadline_alarm_invocations | 10 | 1 num_dedup_inlined_images_candidates_found | 0 | 0 num_dedup_inlined_images_candidates_replaced | 0 | 0 num_fallback_responses_served | 0 | 0 num_fallback_responses_served_while_revalidate | 0 | 0 num_flushes | 831325 | 1 num_js_inlined | 0 | 0 num_local_storage_cache_candidates_added | 0 | 0 num_local_storage_cache_candidates_found | 0 | 0 num_local_storage_cache_candidates_removed | 0 | 0 num_local_storage_cache_stored_css | 0 | 0 num_local_storage_cache_stored_images | 0 | 0 num_local_storage_cache_stored_total | 0 | 0 num_proactively_freshen_user_facing_request | 0 | 0 num_resource_fetch_failures | 15876 | 1 num_resource_fetch_successes | 6478 | 1 num_rewrites_abandoned_for_lock_contention | 2519 | 1 page_load_count | 111093 | 1 pcache_cohorts_beacon_cohort_deletes | 0 | 0 pcache_cohorts_beacon_cohort_hits | 643846 | 1 pcache_cohorts_beacon_cohort_inserts | 92601 | 1 pcache_cohorts_beacon_cohort_misses | 172069 | 1 pcache_cohorts_dependencies_cohort_deletes | 0 | 0 pcache_cohorts_dependencies_cohort_hits | 257790 | 1 pcache_cohorts_dependencies_cohort_inserts | 169473 | 1 pcache_cohorts_dependencies_cohort_misses | 558035 | 1 pcache_cohorts_dom_deletes | 0 | 0 pcache_cohorts_dom_hits | 750667 | 1 pcache_cohorts_dom_inserts | 760324 | 1 pcache_cohorts_dom_misses | 65158 | 1 popularity_contest_num_rewrites_awaiting_retry | 0 | 0 popularity_contest_num_rewrites_running | 0 | 0 popularity_contest_queue_size | 0 | 0 purge_cancellations | 0 | 0 purge_contentions | 0 | 0 purge_file_parse_failures | 0 | 0 purge_file_stats | 0 | 0 purge_file_write_failures | 0 | 0 purge_file_writes | 0 | 0 purge_index | 0 | 0 purge_poll_timestamp_ms | 0 | 0 queued_expensive_operations | 0 | 0 redis_async_deletes | 0 | 0 redis_async_hits | 0 | 0 redis_async_inserts | 0 | 0 redis_async_misses | 0 | 0 redis_blocking_deletes | 0 | 0 redis_blocking_hits | 0 | 0 redis_blocking_inserts | 0 | 0 redis_blocking_misses | 0 | 0 redis_cluster_redirections | 0 | 0 redis_cluster_slots_fetches | 0 | 0 resource_404_count | 3367 | 1 resource_fetch_construct_failures | 26 | 1 resource_fetch_construct_successes | 58 | 1 resource_fetches_cached | 45634 | 1 resource_url_domain_acceptances | 47324330 | 1 resource_url_domain_rejections | 854756 | 1 rewrite_cached_output_hits | 42741964 | 1 rewrite_cached_output_missed_deadline | 40747 | 1 rewrite_cached_output_misses | 8206401 | 1 rewrite_worker_queue_depth | 0 | 0 serf_fetch_active_count | 0 | 0 serf_fetch_bytes_count | 0 | 0 serf_fetch_cancel_count | 0 | 0 serf_fetch_cert_errors | 0 | 0 serf_fetch_failure_count | 0 | 0 serf_fetch_last_check_timestamp_ms | 0 | 0 serf_fetch_request_count | 0 | 0 serf_fetch_time_duration_ms | 0 | 0 serf_fetch_timeout_count | 0 | 0 serf_fetch_ultimate_failure | 0 | 0 serf_fetch_ultimate_success | 0 | 0 shm_cache_deletes | 0 | 0 shm_cache_hits | 0 | 0 shm_cache_inserts | 0 | 0 shm_cache_misses | 0 | 0 show_ads_api_replaced_for_async | 0 | 0 show_ads_snippets_converted | 0 | 0 show_ads_snippets_not_converte | 0 | 0 slurp_404_count | 0 | 0 statistics_404_count | 0 | 0 stdio_fs_outstanding_ops | 0 | 0 stdio_fs_slow_ops | 0 | 0 stdio_fs_total_ops | 0 | 0 successful_downstream_cache_purges | 0 | 0 total_page_load_ms | 522503305 | 1 url_input_resource_hit | 19275 | 1 url_input_resource_miss | 22354 | 1 url_input_resource_recent_fetch_failure | 121 | 1 url_input_resource_recent_uncacheable_failure | 4589 | 1 url_input_resource_recent_uncacheable_miss | 0 | 0 url_trim_saved_bytes | 0 | 0 url_trims | 0 | 0

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pagespeed-ngx/issues/1686#issuecomment-624324400, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAO2IPLYZFQUTDROLC6YMYTRQCCQPANCNFSM4MTB4TGA .

I have no idea what serf is, but yes i'm using nginx with lua.

I tried using this in the URL ?PageSpeedFilters=-in_place_optimize_for_browser and all the PageSpeed links disappeared, site just served standard content.

The whole setup is behind cloudflare.

Here is the full PageSpeed config:

upstream application_server {
    server website-app:8090;
    keepalive 30;
}

include inc/cloudflare.conf;

pagespeed MessageBufferSize 100000;
pagespeed ProcessScriptVariables all;
pagespeed DefaultSharedMemoryCacheKB 0;

server {

    set $root_dir "website";
    set $root_domain "website.com";

    listen 80;
    listen [::]:80;
    listen 443 ssl http2;
    server_name _;

    error_log  /dev/stderr;
    access_log /dev/stdout main;
    root /var/www/$root_dir/public;

    ssl_certificate /etc/nginx/cert.pem;
    ssl_certificate_key /etc/nginx/key.pem;
    ssl_prefer_server_ciphers on;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_session_cache shared:SSL:64m;
    ssl_session_timeout 1h;

    gzip on;
    gzip_vary on;
    gzip_disable "msie6";
    gzip_min_length 256;
    gzip_buffers 16 8k;
    gzip_proxied any;
    gzip_types
        text/plain
        text/css
        text/js
        text/xml
        text/javascript
        font/ttf
        font/opentype
        image/svg+xml
        application/javascript
        application/x-javascript
        application/json
        application/xml
        application/rss+xml
        application/ttf
        application/x-ttf
        application/x-font-otf
        application/font-woff
        application/font-woff2
        application/vnd.ms-fontobject;

    pagespeed on;
    pagespeed FetchHttps disable;
    pagespeed AllowVaryOn "Auto";
    pagespeed EnableCachePurge on;
    pagespeed PurgeMethod PURGE;
    pagespeed SupportNoScriptEnabled false;
    pagespeed InPlaceResourceOptimization on;
    pagespeed LogDir "/var/www/pc/pagespeed/logs/";

    pagespeed LRUCacheKbPerProcess 8192;
    pagespeed LRUCacheByteLimit 16384;

    pagespeed FileCachePath "/var/www/pc/pagespeed/";
    pagespeed FileCacheSizeKb 204800;
    pagespeed FileCacheInodeLimit 5000000;
    pagespeed FileCacheCleanIntervalMs 30000;

    pagespeed MemcachedThreads 2;
    pagespeed MemcachedTimeoutUs 350000;
    pagespeed MemcachedServers "10.1.240.5:11211,10.1.240.6:11211";

    pagespeed CacheFragment $root_dir;
    pagespeed MapOriginDomain http://127.0.0.1 https://$root_domain;
    pagespeed MapOriginDomain http://127.0.0.1 https://stage-00.$root_domain;
    pagespeed MapOriginDomain http://127.0.0.1 https://stage-01.$root_domain;

    pagespeed EnableFilters rewrite_css;
    pagespeed EnableFilters lazyload_images;
    pagespeed EnableFilters remove_comments;
    pagespeed EnableFilters rewrite_javascript;
    pagespeed EnableFilters add_instrumentation;
    pagespeed EnableFilters insert_dns_prefetch;
    pagespeed EnableFilters flatten_css_imports;
    pagespeed EnableFilters collapse_whitespace;
    pagespeed EnableFilters insert_dns_prefetch;
    pagespeed EnableFilters inline_google_font_css;
    pagespeed EnableFilters hint_preload_subresources;
    pagespeed EnableFilters in_place_optimize_for_browser;
    pagespeed EnableFilters canonicalize_javascript_libraries;

    pagespeed EnableFilters rewrite_images;
    pagespeed EnableFilters recompress_webp;
    pagespeed EnableFilters recompress_images;
    pagespeed EnableFilters convert_jpeg_to_webp;
    pagespeed EnableFilters convert_to_webp_lossless;
    pagespeed EnableFilters convert_to_webp_animated;

    pagespeed DisableFilters extend_cache;
    pagespeed DisableFilters extend_cache_images;
    pagespeed DisableFilters responsive_images;
    pagespeed DisableFilters prioritize_critical_css;

    pagespeed DownstreamCachePurgeMethod PURGE;
    pagespeed DownstreamCacheRebeaconingKey "Yhdyf66WWhdy33";
    pagespeed DownstreamCachePurgeLocationPrefix http://127.0.0.1:80;

    pagespeed Disallow "*/admin*";
    pagespeed Disallow "*/healthz";
    pagespeed Disallow "*/ping.js";
    pagespeed Disallow "*/ping.php";
    pagespeed Disallow "*/pixel.gif";
    pagespeed Disallow "*/blank.gif";

    pagespeed Statistics on;
    pagespeed StatisticsLogging on;

    pagespeed StatisticsPath /ngx_pagespeed_statistics;
    pagespeed MessagesPath /ngx_pagespeed_message;
    pagespeed ConsolePath /pagespeed_console;
    pagespeed AdminPath /pagespeed_admin;

    ## Ensure requests for pagespeed optimized resources go to the pagespeed handler
    location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
        add_header "" "";
    }

    location ~ "^/pagespeed_static/" { }
    location ~ "^/ngx_pagespeed_beacon$" { }

    ## PageSpeed Admin UI paths
    location /ngx_pagespeed_statistics {
        include inc/restrictions.conf;
    }
    location /ngx_pagespeed_message {
        include inc/restrictions.conf;
    }
    location /pagespeed_console {
        include inc/restrictions.conf;
    }
    location /pagespeed_admin {
        include inc/restrictions.conf;
    }

    location /proxyHealth {
        pagespeed off;
        empty_gif;
        access_log off;
    }

    location / {
        proxy_pass http://application_server;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header Connection "Keep-Alive";
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

And this is the application nginx:

fastcgi_cache_path /var/lib/nginx/cache/cms-php levels=1:2 keys_zone=cms-php:256m;

include inc/cloudflare.conf;

map $http_cf_ipcountry $user_country {
    default $http_cf_ipcountry;
}

map $request_uri $nocache {
    default 1;
    ~/media 0;
    ~/files 0;
}

map $http_x_forwarded_proto $_https {
    https "on";
}

server {
    listen 0.0.0.0:80;
    server_name _;

    set $root_dir "website";
    set $root_domain "website.com";

    index app.php;
    error_log /dev/stderr;
    access_log /dev/stdout main;
    root /var/www/$root_dir/public;

    rewrite ^/ping.js /ping.php last;
    location ~ ^/ping.php {
        access_log off;
        fastcgi_pass 127.0.0.1:8000;
        include fastcgi_params;
    }

    location /pwa-sw.js {
        access_log off;
        expires max;
        add_header Cache-Control "public, max-age=31557600, s-maxage=31557600";
        add_header Access-Control-Allow-Origin *;
        return 301 $scheme://$root_domain/pwabuilder-sw.js;
    }

    location /images/fb_refer_friend.jpg {
        access_log off;
        expires max;
        add_header Cache-Control "public, max-age=31557600, s-maxage=31557600";
        add_header Access-Control-Allow-Origin *;
        return 301 $scheme://$root_domain/assets/img/frontend/social/fb_open_graph.jpg;
    }

    location ~ ^/apple-touch-icon(-?[0-9x]*)-precomposed\.png$ {
        access_log off;
        expires max;
        add_header Cache-Control "public, max-age=31557600, s-maxage=31557600";
        add_header Access-Control-Allow-Origin *;
        return 301 $scheme://$root_domain/apple-touch-icon$1.png;
    }

    location /assets {
        access_log off;
        expires max;
        add_header Cache-Control "public, max-age=31557600, s-maxage=31557600";
        add_header Access-Control-Allow-Origin *;
    }

    location ~ ^/(css|fonts|img|js|vendor) {
        access_log off;
        expires max;
        add_header Cache-Control "public, max-age=31557600, s-maxage=31557600";
        add_header Access-Control-Allow-Origin *;
        root /var/www/$root_dir/public/assets;
    }

    location ~ ^/(media|files) {
        access_log off;
        expires max;
        add_header Cache-Control "public, max-age=31557600, s-maxage=31557600";
        add_header Access-Control-Allow-Origin *;
        try_files $uri /app.php$is_args$args;
    }

    location /admin {
        access_log off;
        include inc/restrictions.conf;
        try_files $uri /app.php$is_args$args;
    }

    location /healthz {
        access_log off;
        include inc/restrictions.conf;
        try_files $uri /app.php$is_args$args;
    }

    location /http-cache {
        access_log off;
        deny all;
    }

    location / {
        try_files $uri /app.php$is_args$args;
    }

    location ~ ^/(ping|app)\.php(/|$) {
        set_by_lua_block $ismobile {
            local ismobile = mobile_detect.mobile(ngx.var.http_user_agent or "")
            if ismobile then
                return "mobile"
            end
            return "desktop"
        }

        log_by_lua_block {
            metric_requests:inc(1, {ngx.var.hostname, ngx.var.status})
            metric_latency:observe(ngx.now() - ngx.req.start_time(), {ngx.var.hostname})
        }

        fastcgi_pass 127.0.0.1:8000;
        include fastcgi_params;

        fastcgi_cache cms-php;
        fastcgi_cache_min_uses 1;
        fastcgi_cache_valid 200 336h;
        fastcgi_cache_valid 404 0m;
        fastcgi_cache_valid 500 0m;
        fastcgi_cache_valid 301 0m;
        fastcgi_cache_use_stale updating error timeout invalid_header http_500;
        fastcgi_cache_purge PURGE from all;
        fastcgi_cache_key $request_uri;
        fastcgi_cache_bypass $nocache;
        fastcgi_no_cache $nocache;
    }

    location ~ blank\.gif$ {
        empty_gif;
        expires off;
        access_log off;
        add_header Cache-Control "public, max-age=31557600, s-maxage=31557600";
    }

    location ~ \.php$ {
        deny all;
    }

    location ~ /\..* {
        deny all;
    }
}

I'm drawn to the contrast between this stanza in your nginx.conf:

ssl_certificate /etc/nginx/cert.pem;
ssl_certificate_key /etc/nginx/key.pem;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers

EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5; ssl_session_cache shared:SSL:64m; ssl_session_timeout 1h;

And this line from the PageSpeed section:

pagespeed FetchHttps disable;

Are you serving your content via HTTPS? Can you switch that 'disable' to an 'enable'? And then see what happens to the serf stats? Yours are currently zero.

serf_fetch_active_count | 0 | 0 serf_fetch_bytes_count | 0 | 0 serf_fetch_cancel_count | 0 | 0 serf_fetch_cert_errors | 0 | 0 serf_fetch_failure_count | 0 | 0 serf_fetch_last_check_timestamp_ms | 0 | 0 serf_fetch_request_count | 0 | 0 serf_fetch_time_duration_ms | 0 | 0 serf_fetch_timeout_count | 0 | 0 serf_fetch_ultimate_failure | 0 | 0 serf_fetch_ultimate_success | 0 | 0

Serf, by the way, is the HTTP fetching client library used by mod_pagespeed. https://serf.apache.org/ . In nginx there is also a 'native' option: https://www.modpagespeed.com/doc/system#native-fetcher, but I don't think it's on by default. Just in case I'm not remembering correctly, can you also add:

pagespeed UseNativeFetcher off;

On Tue, May 5, 2020 at 6:04 PM Andrew Borg [email protected] wrote:

I have no idea what serf is, but yes i'm using nginx with lua.

I tried using this in the URL ?PageSpeedFilters=-in_place_optimize_for_browser and all the PageSpeed links disappeared, site just served standard content.

Here is the full PageSpeed config:

upstream application_server { server website-app:8090; keepalive 30; }

include inc/cloudflare.conf;

pagespeed MessageBufferSize 100000; pagespeed ProcessScriptVariables all; pagespeed DefaultSharedMemoryCacheKB 0;

server {

set $root_dir "website";
set $root_domain "website.com";

listen 80;
listen [::]:80;
listen 443 ssl http2;
server_name _;

error_log  /dev/stderr;
access_log /dev/stdout main;
root /var/www/$root_dir/public;

ssl_certificate /etc/nginx/cert.pem;
ssl_certificate_key /etc/nginx/key.pem;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
ssl_session_cache shared:SSL:64m;
ssl_session_timeout 1h;

gzip on;
gzip_vary on;
gzip_disable "msie6";
gzip_min_length 256;
gzip_buffers 16 8k;
gzip_proxied any;
gzip_types
    text/plain
    text/css
    text/js
    text/xml
    text/javascript
    font/ttf
    font/opentype
    image/svg+xml
    application/javascript
    application/x-javascript
    application/json
    application/xml
    application/rss+xml
    application/ttf
    application/x-ttf
    application/x-font-otf
    application/font-woff
    application/font-woff2
    application/vnd.ms-fontobject;

pagespeed on;
pagespeed FetchHttps disable;
pagespeed AllowVaryOn "Auto";
pagespeed EnableCachePurge on;
pagespeed PurgeMethod PURGE;
pagespeed SupportNoScriptEnabled false;
pagespeed InPlaceResourceOptimization on;
pagespeed LogDir "/var/www/pc/pagespeed/logs/";

pagespeed LRUCacheKbPerProcess 8192;
pagespeed LRUCacheByteLimit 16384;

pagespeed FileCachePath "/var/www/pc/pagespeed/";
pagespeed FileCacheSizeKb 204800;
pagespeed FileCacheInodeLimit 5000000;
pagespeed FileCacheCleanIntervalMs 30000;

pagespeed MemcachedThreads 2;
pagespeed MemcachedTimeoutUs 350000;
pagespeed MemcachedServers "10.1.240.5:11211,10.1.240.6:11211";

pagespeed CacheFragment $root_dir;
pagespeed MapOriginDomain http://127.0.0.1 https://$root_domain;
pagespeed MapOriginDomain http://127.0.0.1 https://stage-00.$root_domain;
pagespeed MapOriginDomain http://127.0.0.1 https://stage-01.$root_domain;

pagespeed EnableFilters rewrite_css;
pagespeed EnableFilters lazyload_images;
pagespeed EnableFilters remove_comments;
pagespeed EnableFilters rewrite_javascript;
pagespeed EnableFilters add_instrumentation;
pagespeed EnableFilters insert_dns_prefetch;
pagespeed EnableFilters flatten_css_imports;
pagespeed EnableFilters collapse_whitespace;
pagespeed EnableFilters insert_dns_prefetch;
pagespeed EnableFilters inline_google_font_css;
pagespeed EnableFilters hint_preload_subresources;
pagespeed EnableFilters in_place_optimize_for_browser;
pagespeed EnableFilters canonicalize_javascript_libraries;

pagespeed EnableFilters rewrite_images;
pagespeed EnableFilters recompress_webp;
pagespeed EnableFilters recompress_images;
pagespeed EnableFilters convert_jpeg_to_webp;
pagespeed EnableFilters convert_to_webp_lossless;
pagespeed EnableFilters convert_to_webp_animated;

pagespeed DisableFilters extend_cache;
pagespeed DisableFilters extend_cache_images;
pagespeed DisableFilters responsive_images;
pagespeed DisableFilters prioritize_critical_css;

pagespeed DownstreamCachePurgeMethod PURGE;
pagespeed DownstreamCacheRebeaconingKey "Yhdyf66WWhdy33";
pagespeed DownstreamCachePurgeLocationPrefix http://127.0.0.1:80;

pagespeed Disallow "*/admin*";
pagespeed Disallow "*/healthz";
pagespeed Disallow "*/ping.js";
pagespeed Disallow "*/ping.php";
pagespeed Disallow "*/pixel.gif";
pagespeed Disallow "*/blank.gif";

pagespeed Statistics on;
pagespeed StatisticsLogging on;

pagespeed StatisticsPath /ngx_pagespeed_statistics;
pagespeed MessagesPath /ngx_pagespeed_message;
pagespeed ConsolePath /pagespeed_console;
pagespeed AdminPath /pagespeed_admin;

## Ensure requests for pagespeed optimized resources go to the pagespeed handler
location ~ "\.pagespeed\.([a-z]\.)?[a-z]{2}\.[^.]{10}\.[^.]+" {
    add_header "" "";
}

location ~ "^/pagespeed_static/" { }
location ~ "^/ngx_pagespeed_beacon$" { }

## PageSpeed Admin UI paths
location /ngx_pagespeed_statistics {
    include inc/restrictions.conf;
}
location /ngx_pagespeed_message {
    include inc/restrictions.conf;
}
location /pagespeed_console {
    include inc/restrictions.conf;
}
location /pagespeed_admin {
    include inc/restrictions.conf;
}

location /proxyHealth {
    pagespeed off;
    empty_gif;
    access_log off;
}

location / {
    proxy_pass http://application_server;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header Connection "Keep-Alive";
    proxy_set_header X-Forwarded-For $remote_addr;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Port $server_port;
    proxy_set_header X-Forwarded-Proto $scheme;
}

}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pagespeed-ngx/issues/1686#issuecomment-624331129, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAO2IPIL3RMC7CO3OQZSKVDRQCEPFANCNFSM4MTB4TGA .

Yes, the website serves content over HTTPS, however FetchHttps would not work, the PageSpeed Nginx is the one which is internet facing and terminates SSL. Then it forwards the request to the application Nginx, which only listens over HTTP. That is why i disabled https for PageSpeed and set the MapOriginDomain to 127.0.0.0 over port 80.

I'll try disabling the UseNativeFetcher.

Quick update, I've disabled InPlaceOptimization and also changed the MapOriginDomain to use the kubernetes internal DNS rather than 127.0.0.1.

If there was the issue that PageSpeed could not reach the assets now that is gone. Assets got re-optimised and all is good so far.

I'll keep an eye on the cache DB, theoretically the memory used should level out (or at least not grow as fast). I'll keep u guys updated :)

Thanks for everything :)

Awesome. I think now you should be able to re-enable InPlaceResourceOptimization. And great job getting the fetch-origin sorted out; that's a great remedy.

On Thu, May 7, 2020 at 2:39 PM Andrew Borg [email protected] wrote:

Quick update, I've disabled InPlaceOptimization and also changed the MapOriginDomain to use the kubernetes internal DNS rather than 127.0.0.1.

If there was the issue that PageSpeed could not reach the assets now that is gone. Assets got re-optimised and all is good so far.

I'll keep an eye on the cache DB, theoretically the memory used should level out (or at least not grow as fast). I'll keep u guys updated :)

Thanks for everything :)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/apache/incubator-pagespeed-ngx/issues/1686#issuecomment-625428537, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAO2IPMRRW6ADFG5PKPPHODRQL57PANCNFSM4MTB4TGA .

Thanks.

I think for the next week or so i'll just leave it off, see what happens with the cache size. The image are re-written now anyways so the site is fine :)