Cached 503s for static files

[florisschreuder]

Hi,

With a client of ours we had an incident last saturday where several (it seemed to switch which) Javascript and image files gave a 503 response from Fastly. Which is strange - 503s are generally either for PHP-runtime-errors or an origin which is entirely unresponsive. Strangely, it seemed to fluctuate by client which files were getting a 503. When disabling my VPN, the files which got 503s switched.

We put Fastly on a 'pass all' and got to work on it this morning. It seems we had an 'attack' of some sort last friday which temporarily took the origin server offline/very slow. The 503s started then and continued a full 24 hours until we put in the 'pass all'.

What eventually worked today is that we removed the static maintenance page that the Magento 2 module created.

I could be (very) wrong, but is it possible that these lines created the problem?

• https://github.com/fastly/fastly-magento2/blob/707ffb3779a1d3c7921b81a88a69c5613490a702/Controller/Adminhtml/FastlyCdn/SyntheticPages/SaveErrorPageHtml.php#L117-L122
• https://github.com/fastly/fastly-magento2/blob/master/etc/vcl_snippets/recv.vcl#L22-L25

Static files seem like they are excluded here (in de recv.vcl), however, the URL of a pub/static file is (generally) not pub/static, but just static. So if the static file isn't excluded but had gotten a 503 in the past and is 'stuck' on another edge server, the 'ResponseObject' is set to '970' and the static maintenance page is served.

I'm not fluent enough in Varnish VCLs or Fastly to be sure, but I figured I'd report this so you could check the logic.

Thanks in advance!

[vvuksan]

Hi @florisschreuder. It is true that static file will indeed get the static maintenance page URL served in case of a 500 error (which also covers things like 502 Gateway errors, etc). That said 503s are not cached by default (you would need to have VCL logic that makes it cacheable).

[florisschreuder]

Hi @vvuksan ,

Thanks for the quick response!

Hmm, I wonder what the hell caused it. As mentioned, it all started with an 'attack' by a bad-actor trying to SQL inject or something. Weirder still is that when changing IPs, the files that returned a 503 also changed - like different edge-servers had this problem. Our problem was fixed after we changed to the 503-page in the Fastly.com UI instead of the one with the Magento2 module snippet.

But in either case: the if-statement in the snippet should be changed from

        !req.http.Fastly-Client-Ip ~ maint_allowlist &&
        !req.url ~ "^/(index\.php/)?####ADMIN_PATH####/" &&
        !req.url ~ "^/pub/static/") {

To

        !req.http.Fastly-Client-Ip ~ maint_allowlist &&
        !req.url ~ "^/(index\.php/)?####ADMIN_PATH####/" &&
        !req.url ~ "^/(pub/)?static/") {

Right?