bug: response-rewrite plugin will lose Content-Encoding when the body returned by upstream has compression encoding

[tangzhenhuang]

Current Behavior

Content-Encoding will be lost when upstream returns body with compression encoding.

Expected Behavior

Keep Content-Encoding

Error Logs

When debugging on the browser side, you will find that the Content-Encoding header will be lost after the plugin is enabled. If you force the Content-Encoding: br header, it will be normal.

Steps to Reproduce

1. upstream returns body with compression encoding, for example :https://github.com/koajs/compress
2. Create a route with a response-rewrite plugin that uses the filters field to rewrite any responseBody
3. access by chrome

Environment

• APISIX version (run apisix version): 2.15
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

[tangzhenhuang]

I guess that the Content-Encoding is lost when calling core.response.holdbody chunk(ctx). By the way, I debugged this method and found that the body read by this method is not decoded (if the upstream Compression encoding)

[tangzhenhuang]

I found that the header_filter of the plugin will call the core.response.clear_header_as_body_modified() method to clear the relevant headers :-(

[tzssangglass]

I found that the header_filter of the plugin will call the core.response.clear_header_as_body_modified() method to clear the relevant headers :-(

Yes, because if the response is modified by response-rewrite plugin, the origin Content-Length in the response header may be invalid, so need to clean the headers that are ready to send to the client.

[tzssangglass]

It look like we need to make response-rewrite support:

1. decode upstream body by Content-Encoding form upstream;
2. rewrite body
3. encode body with the origin Content-Encoding
4. add Content-Encoding to headers when rebuild headers for client

I think this is not a bug but a feature.

[tokers]

@crazyMonkey1995 Your rewrite should fail since Response Rewrite plugin doesn't decode the response body from upstream.

[tokers]

It look like we need to make response-rewrite support:

1. decode upstream body by Content-Encoding form upstream;
2. rewrite body
3. encode body with the origin Content-Encoding
4. add Content-Encoding to headers when rebuild headers for client

I think this is not a bug but a feature.

That would be tough if support it only on Lua level.

[tangzhenhuang]

@crazyMonkey1995 Your rewrite should fail since Response Rewrite plugin doesn't decode the response body from upstream.

Indeed, it fails to rewrite, which is equivalent to doing nothing, but removes Content-Encoding, so browser-side parsing fails，or can we do this: judge Content-Encoding in the rewrite header stage, if there is no way to rewrite the body, do nothing

[tzssangglass]

Indeed, it fails to rewrite, which is equivalent to doing nothing, but removes Content-Encoding, so browser-side parsing fails，or can we do this: judge Content-Encoding in the rewrite header stage, if there is no way to rewrite the body, do nothing

So why do we need response-rewrite? I probably understand that the response-rewrite plugin is configured on the route, but some of the requests matching this route do not need to execute the response-rewrite plugin.

we can make the filter in _meta( https://github.com/apache/apisix/blob/master/docs/en/latest/terminology/plugin.md#dynamically-control-whether-the-plugin-is-executed) to support dynamically executed by Content-Encoding form upstream.

[spacewander]

we can make the filter in _meta

Before that, we need to make _meta.filter run after receiving the response. Currently, it is run only once after the plugin configuration is merged.

[tzssangglass]

I probably understand that the response-rewrite plugin is configured on the route, but some of the requests matching this route do not need to execute the response-rewrite plugin.

now we can use vars in response-rewrite plugin to do something like this