ETag vs Memory Page Cache

[viluon]

I want to use both entity tag headers and Lapis' server-side cache. The latter for avoiding re-renders, the former for avoiding re-downloads. However, when using Lapis' page cache, I cannot modify the headers sent to the client when a cache hit happens. Consider the following code:

app:match("article", "/article/:name", capture_errors {
	on_error = function()
		return {
			render = "article-not-found";
			status = 404;
		}
	end,
	cached {
		exptime = DEV and 1 or 3600;	-- 1 hour
		function(self)
			local start = os.clock()
			self.article = require "lib.article"
			self.yield_error = yield_error

			local log = self.article.get_log(self.params.name)
			local sum = self.article.get_sum(self.params.name)
			local heading = self.article.get_heading(self.params.name)

			self.log = log
			self.heading = heading
			self.page_title = (#log ~= 0 and heading) and (heading .. " by " .. log[#log][1]) or nil;

			return {
				render = true;
				headers = {
					["x-I-love"] = "cookies";
					["etag"] = '"' .. sum .. '"';
					["x-lua-delay-ms"] = (os.clock() - start) * 1000;
				}
			}
		end
	}
})

The first request to an article resource saves the result in the page cache, as intended. These are the response headers for the first request:

With subsequent requests, however, the headers of the original response aren't included:

This results in a 200/OK status code rather than a 304/Not Modified. The client is forced to re-download the resource, even though it hasn't changed.

This means I have two options:

1. I could disable the server-side cache, which would make ETags work (and would avoid useless downloads for clients), but the server would be rendering the same page over and over for multiple clients, needlessly
2. I could disable the ETags and use only the server-side cache. That'd mean the server is more efficient, but clients need to download resources over and over (this is what happens now, because ETag headers are missing in cached responses)

I'd like to combine the two approaches, so that renders are cached and no unnecessary downloads happen. For static assets there's no rendering going on, so I just use an expires directive in nginx.conf.

EDIT: A solution could be as simple as adding an optional preserved_headers field to the table passed to the cached() function (somewhere around here) which would be an array or a lookup table of headers stored in the cache entry alongside the response body.

EDIT2: Actually, that may not work. I don't know how exactly does nginx decide on whether to send a 304 or not, but if the Lapis code forces a cache load, it'll probably bypass nginx's ETag check. A solution would thus be checking the ETag on the Lua (Lapis) side of things in the cache hit code. That's a bit ugly though, I'd rather let nginx handle it all...

[leafo]

Sorry for taking forever get back to you.

I just checked over the lapis cache code and it doesn't save any response headers other than content type right now. I would consider this a bug. The memory cache is very basic.

For my own work I've switched to using nginx's built in cache with the proxy module using a proxy-to-self technique. I've found that the built in cache is very powerful: it can hold significantly more since it offloads to disk, it's still very fast, you can skip lapis entirely on cache hit. I have a WIP blog post here outlining how it works: http://leafo.net/guides/nginx-reverse-proxy-to-self.html#enhanced-configuration/using-the-nginx-caching-module

You can see an implementation of this on the luarocks repo: https://github.com/luarocks/luarocks-site/blob/master/nginx.conf#L38