doc.rust-lang.org: add caching headers

[jsha]

(moving from https://github.com/rust-lang/rust/issues/82286)

Steps to reproduce:

1. Load any doc.rust-lang.org page, e.g. https://doc.rust-lang.org/std/str/index.html.
2. Open Developer Console.
3. Click the URL bar and hit enter to load the page again.

Expected result:

Most resources are cached and do not hit the network.

Actual result:

Many objects require a network fetch, but get a 304 Not Modified. These should be served with long caching headers.

Current headers:

$ curl -I https://doc.rust-lang.org/SourceCodePro-Regular.woff
HTTP/2 200 
content-type: font/woff
content-length: 55472
last-modified: Thu, 11 Feb 2021 14:27:46 GMT
x-amz-version-id: R.xOjlR7eKEsQwzm9EIrHs9eeCIDeXYq
server: AmazonS3
date: Thu, 18 Feb 2021 13:27:31 GMT
etag: "957fa8c030f8116bea59c13df470e4e8"
x-cache: Hit from cloudfront
via: 1.1 4b84530d7a095b58fb7a1d20b7f0cbe0.cloudfront.net (CloudFront)
x-amz-cf-pop: HIO50-C2
x-amz-cf-id: QpMYsW6BVcTPpDmi8Pg9HiMz_1iSOIpdYv60MMzNqVoV707yqsv_Dw==
age: 63023

[Nemo157]

All the files with the version numbers embedded in them, or files hosted under the versioned doc paths (e.g. https://doc.rust-lang.org/1.50.0/std/) seem like they could be marked as cache-control: immutable (and all the other headers that go with it). That still leaves the font-files on the root/named docs as not really being cacheable since they may change when a release is made; maybe these files should have the fonts version number embedded in their filename?

[jsha]

I think it would be reasonable to set the fonts to have a TTL of a few weeks and not worry about versioning them. They're not required to be strongly consistent with the rest of the content. If the fonts do change, and it takes someone a while to get the new version, no big deal.

[jsha]

A friendly bump on this. I think it would make a noticeable improvement to the experience of people reading docs on rust-lang.org, particularly if they have slow connections.

[jsha]

Did some more thinking and research in this Zulip thread: https://rust-lang.zulipchat.com/#narrow/stream/266220-rustdoc/topic/caching.20for.20doc.2Erust-lang.2Eorg.20docs/near/262957610

---

For the stdlib docs on doc.rust-lang.org, here's the current situation for caching: everything is in S3, so it gets ETags and Last-Modified automatically. The Cache-Control headers are controlled by what's set when uploading to S3, which is currently nothing. That means it's up to browsers to apply a heuristic about how long to cache.

According to https://paulcalvano.com/2018-03-14-http-heuristic-caching-missing-cache-control-and-expires-headers-explained/, that heuristic is "10% of the time since Last-Modified." So for anything in the current stable docs, released 29 days ago, the heuristic cache time is 2.9 days. After the next stable is released, the heuristic cache time will be ~0 days, and slowly go up.

It'd be nice to do better! In particular, for versioned static assets like CSS, JS, and fonts (especially fonts), we'd like to do as docs.rs does and set a long max-age plus "immutable." For nightly, this doesn't really work, since those static assets get overwritten (with the same version number) every night. However, for the stable docs, this could work.

For HTML in the stable docs: We could set "Expires" to the scheduled release of the next stable version. But that would create a problem when there needs to be a patchlevel release in between. During an uneventful 6-week release cycle with no patchlevel releases, the heuristic cache time would get as high as 4.2 days. So we could safely set max-age=4.2 days and have not much worse than the current situation. Or we could round up a bit in the name of more caching and say 7 days. We could also set stale-while-revalidate to some period on the order of months.

For HTML in the nightly docs: Here we can quite confidently set Expires to the time of the next planned nightly build, since it's unlikely there will be another build in between.

---

Here's my proposal:

• For nightly uploads, set the Expires header to the typical deploy time the next night.
• For beta uploads, 1 week.
• For stable uploads, set the Expires header to the next scheduled release date (assume the regular 6 week cycle).
• For uploads to versioned paths (https://doc.rust-lang.org/1.56.0/), set Cache-Control: max-age=1 year, immutable.
• For fonts, set Cache-Control: max-age=1 year, immutable.