robot.txt, favicons cached for 1 year by default in production

[zarqman]

Summary: Non-digested assets (including those outside /public/assets) default to the same 1 year cache period in cache-control as digested assets. Since non-digested assets are mutable, a 1 year cache period is inappropriate.

Background

The built-in public file server's current default cache-control specifies a 1 year cache period in production.rb. While this is fine for digested assets, it also applies to non-digested assets such as robots.txt, favicons, and anything else in /public, even if not in /public/assets.

https://github.com/rails/rails/blob/a8709e6ea26eca73a652af4fdd0a9f7db5352af4/railties/lib/rails/generators/rails/app/templates/config/environments/production.rb.tt#L20-L21

The comment notes that "they are all digest stamped", but indeed this is not the case.

To make Rails friendly to folks who are not well versed in HTTP caching semantics, I believe the default period in production.rb should be radically shorter. I would suggest 1 day maximum, and perhaps just 6 to 12 hours. This would be substantially more forgiving to mistakes than being stuck with something cached in a remote system for days or weeks.

Favicons in particular can be cached for extended periods. Crawlers are unlikely to honor a cached robots.txt for a full year, but are likely to honor the cache period for several days into weeks.

Changing this would affect digested assets too, but I suggest that there's little practical difference in overhead for the server between 1 day and 1 year. I'd suggest dropping the default period in development and test as well, to match, but those aren't particularly critical in practice.

Would there be any openness to such a change? I'm happy to create a PR if so (or someone else can--either way).

Steps to reproduce

In production:

curl -I http://some-production-app.com/robots.txt

In development:

# Enable caching
rails dev:cache
curl -I http://localhost:3000/icon.svg

Note that the default in development is 2 days, so that's what will be returned.

Expected behavior

The default cache period for non-digested static files should be forgiving to changes and mistakes.

I'd suggest a default cache period of something between 6 and 24 hours.

Actual behavior

Max age is 1 year for a non-digested URL. Note that both robots.txt and icon.svg are default, stock files that are generated by a newly generated Rails app.

Production:

$ curl -i http://some-production-app.com/robots.txt
HTTP/1.1 200 OK
Cache-Control: public, max-age=31556952
Content-Length: 99
Content-Type: text/plain
Last-Modified: Tue, 19 Nov 2024 21:53:41 GMT
Date: Wed, 20 Nov 2024 18:08:41 GMT

Development (with rails dev:cache):

$ curl -I http://localhost:3000/icon.svg
HTTP/1.1 200 OK
last-modified: Wed, 20 Nov 2024 01:56:36 GMT
content-type: image/svg+xml
cache-control: public, max-age=172800
Content-Length: 122

System configuration

Rails version: 8.0.0

Ruby version: 3.3.6

[rails-bot[bot]]

This issue has been automatically marked as stale because it has not been commented on for at least three months. The resources of the Rails team are limited, and so we are asking for your help. If you can still reproduce this error on the 8-0-stable branch or on main, please reply with all of the information you have about it in order to keep the issue open. Thank you for all your contributions.

[zarqman]

still an issue

[rails-bot[bot]]

This issue has been automatically marked as stale because it has not been commented on for at least three months. The resources of the Rails team are limited, and so we are asking for your help. If you can still reproduce this error on the 8-0-stable branch or on main, please reply with all of the information you have about it in order to keep the issue open. Thank you for all your contributions.

[zzak]

I think GitHub auto-closed this because the message had "Fix #NNNNN" but was apart of the message for the revert.