"Cache-Control" header and "Age" header are removed by ESI module

[david187]

Version:

ATS 9.2.0

Reproduction

Master Document Header: X-Esi: 1 etag: W/"xxxxx" content-type: text/html; charset=UTF-8 Cache-Control: max-age=30

ESI sub-document 1 header: cache-control: public, max-age=30 content-type: text/html; charset=UTF-8 etag: W/"XXXXXX"

ESI sub-document 2 header: cache-control: public, max-age=30 content-type: text/html; charset=UTF-8 etag: W/"XXXXXX"

Final Output header:

content-type: text/html; charset=UTF-8 etag: W/"XXXXXX" X-Cache-Status: UPDATED

request after 10s content-type: text/html; charset=UTF-8 etag: W/"XXXXXX" X-Cache-Status: HIT

so, the caching seem work internally. but the headers are removed.

The "Cache-Control" header and "Age" header are removed by ESI module

Expected Results

Final Output header: content-type: text/html; charset=UTF-8 etag: W/"XXXXXX" cache-control: public, max-age=30 Age: 0

and, if request after 10s

content-type: text/html; charset=UTF-8 etag: W/"XXXXXX" cache-control: public, max-age=30 Age: 10

The cahe control / Age header should be keep to let upper layer / client cache work.

Here are some code about the max-age header calculation, need addition config to make it take effect ？ https://github.com/apache/trafficserver/blob/0e9ff8271811a46b46825fe22a272347b64df078/plugins/esi/combo_handler.cc#L266

[shukitchan]

Yes. the cache control header is removed - https://github.com/apache/trafficserver/blob/master/plugins/esi/esi.cc#L1111

We can either add an option to stop it from happening. Or you can use header_rewrite / lua plugin to add the cache control header back at the "send response header" hook.

[david187]

The reason of remove these headers is?

I can access and copy the "Cache-Control" header as "X-Cache-Control" during TS_LUA_HOOK_READ_RESPONSE_HDR hook. However, if i restore the "X-Cache-Control" as "Cache-Control" during TS_LUA_HOOK_SEND_RESPONSE_HDR. it will be removed. ( Although I can restore at the upper layer)

However, the age header seem can not access by ts.cached_response.header / ts.server_response.header in lua plugin. I think it is useful for multiple layer caching.

Temporarily, I am trying to add a timestamp header to calculate the age.

And the other question is, can I access the sub-document's headers in the master document by lua ?

===

We can either add an option to stop it from happening. can add a new options at next version?

=========

update: finally, I get the age header during TS_LUA_HOOK_SEND_RESPONSE_HDR by using: ts.client_response.header['Age']

[shukitchan]

I can't quite remember why we did that in the first place. Perhaps we want to later add a private cache control header.

ESI plugin added a transaction hook to remove these headers - https://github.com/apache/trafficserver/blob/master/plugins/esi/esi.cc#L1426

So if you use lua, you need to have the lua plugin after ESI plugin and also adds a transaction hook for TS_LUA_HOOK_SEND_RESPONSE_HDR at the do_global_send_response() function

Lua is not aware of ESI at all so there is no way to retrieve the sub document headers.

We can add a new options at next version or any contributions are welcome as well.

[david187]

Is it possible to keep specify sub-document headers( such as etag, cache-control, last-modified ) somewhere.

or combo the specify headers as "X-ESI-" headers: X-ESI-Etags: , <sub-document etag 1>, <sub-document etag 2>... X-ESI-Last-Modified: X-ESI-Cache-Control: <public | private>, max-age=

https://github.com/apache/trafficserver/blob/0e9ff8271811a46b46825fe22a272347b64df078/plugins/esi/combo_handler.cc#L266

It will be useful for parent / multiple layer caching.

[shukitchan]

Theoretically yes. The changes will be quite a lot, though.

Alternatively, lua plugin has a fetch function that returns the headers of the fetch URL - https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/lua.en.html#ts-fetch

Of course the downside of it is that you need to implement the ESI-like content replacement on your own inside the lua script as well.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.