ngx.resp.get_headers(): Why is the response header incomplete?

[chenshanyi]

config: location =/version { default_type 'text/plain';

        content_by_lua_file /usr/local/lmrs/lua/api/version.lua;
    
        log_by_lua '
	ngx.log(ngx.ERR,"header: ", require"cjson".encode(ngx.resp.get_headers()))
    ';
    }

request: curl -i "http://127.0.0.1:8088/version" HTTP/1.1 200 OK Server: nginx/1.12.2 Date: Tue, 10 Sep 2019 13:17:46 GMT Content-Type: text/plain Transfer-Encoding: chunked Connection: keep-alive

log: 2019/09/10 09:12:14 [error] 123144#0: *3 [lua] log_by_lua(lmrs.conf:195):2: header: {"transfer-encoding":"chunked","content-type":"text/plain","connection":"close"} while logging request, client: 127.0.0.1, server: , request: "GET /version HTTP/1.1", host: "127.0.0.1:8088"

Why is the response header incomplete?

[dndx]

Hello,

Nginx in general stores headers in a linked list. But this has a drawback of changing and accessing a header requires traversing the list to find it which may be slow. As an optimization Nginx also defined a bunch of "commonly used" headers that are not stored in the list, but rather in the struct itself, this guarantees O(1) access while changing/reading those "commonly used" headers.

See definitions of ngx_http_headers_out_t struct

ngx.resp.get_headers calls ngx_http_lua_ngx_resp_get_headers. In that function we did not handle all the cases for "fast path" headers. AFAIK only content-type, content-length, connection and transfer-encoding were handled.

From Nginx source that handles $sent_http_* variables, the following header support needs to be added for ngx.resp.get_headers to behave correctly:

• Location
• Last-Modified
• Keep-Alive
• Cache-Control
• Link
• Date
• Server

Note that the last two headers, Date and Server were not even handled by the $sent_http_* variables properly, presumably they were overlooked. This was confirmed with my local testing by reading $sent_http_server and $sent_http_date in log phase. Both returned nil.

[ghost]

Is there any work around for this? Our own requirements are to log in access logs the Location response header which works simply with $upstream_http_location. However we have additional requirements to mask out some secure data from the parameters in that url. I have some lua script that does that masking successfully but we can't pass it ngx.resp.get_headers()["location"].

[bjuergens]

I don't know much about nginx or lua, but I noticed some weird behavior, which let me to find this issue.

I was playing around with luafilters for headers, when I noticed, that ngx.resp.get_headers gives a different number of headers when called from set_by_lua_block and from header_filter_by_lua

consider the following example

set_by_lua_block $resp_headers_short{
    local lua_resp_headers_short = ""
    for k, v in pairs(ngx.resp.get_headers()) do
        lua_resp_headers_short = lua_resp_headers_short .. k.."="..v.." "
    end
    return lua_resp_headers_short
}

set $resp_headers_full "";
header_filter_by_lua '
    for k, v in pairs(ngx.resp.get_headers()) do
        ngx.var.resp_headers_full = ngx.var.resp_headers_full .. k.."="..v.." "
    end
';

the first block only produces connection=keep-alive, while the latter block produces all of them last-modified=Tue, 13 Apr 2021 15:13:59 GMT etag=\"6075b537-264\" accept-ranges=bytes content-type=text/html connection=keep-alive content-length=612 (sans date and server)

considering the comment above, I guess that header_filter_by_lua is executed at a different time, where fast-path headers are already part of the request.

Is there any work around for this?

use header_filter_by_lua, apparently

[alexdowad]

Is there any work around for this? Our own requirements are to log in access logs the Location response header which works simply with $upstream_http_location. However we have additional requirements to mask out some secure data from the parameters in that url. I have some lua script that does that masking successfully but we can't pass it ngx.resp.get_headers()["location"].

I had the same problem of trying to access the Location header in a log_by_lua block (though a different use case from you). @dndx's comment gave me the needed clue.

I was able to get it with ngx.var.sent_http_location. I guess for you it would be ngx.var.upstream_http_location.