X-Accel-Redirect style of header matching is broken in handle_response

[coolaj86]

Caddy v2.7.6

It seems that rewrite will only work in handle_response if copy_response_headers is set with the same variable that was used in the rewrite (i.e. X-Accel-Redirect must be exposed downstream).

Instead of getting the files as expected, I get 404s.

+             copy_response_headers {
+                 include X-Accel-Redirect
+             }

              rewrite * {http.response.header.X-Accel-Redirect}

I stumbled upon the workaround because I was using copy_response_headers to debug and sanity check that the header I was setting was being seen by Caddy.

Then the rewrite suddenly started working.

I take that away and it 404s again.

Additionally, I can't use header_down -X-Accel-Redirect to get rid of the of copied header.

Failing

https://localhost {
    reverse_proxy /api/* http://localhost:3001 {
        @sendfile header X-Accel-Redirect *
        handle_response @sendfile {
            rewrite * {http.response.header.X-Accel-Redirect}
            root ./private/
            file_server
        }
    }
}

Workaround

https://localhost {
    reverse_proxy /api/* http://localhost:3001 {
        @sendfile header X-Accel-Redirect *
        handle_response @sendfile {
            copy_response_headers {
                include X-Accel-Redirect
            }

            rewrite * {http.response.header.X-Accel-Redirect}
            root ./private/
            file_server
        }
    }
}

Best Guess

My bet is that the priority of when http.response.header.Whatever is set has changed such that it refers to the final headers to the Caddy client rather than the intermediary headers from the reverse proxy.

copy_response_headers causes the header to be set before rewrite takes place.

I suppose that the priority currently is:

• header_down hence causes neither failure nor removes the header
• copy_response_headers makes the header available to rewrite
• rewrite can't access the response header if it hasn't been copied to the response

Full Caddyfile

I doubt the other directives I have are at play since I've identified exactly which change works and fails, but just in case there's a strange interaction, this is very, very close to my actual Caddyfile:

https://localhost {
    tls internal
    root * ./dist/
    file_server
    try_files {path} {path}.html
    reverse_proxy /api/* http://localhost:3001 {
        @sendfile header X-Accel-Redirect *
        handle_response @sendfile {
            # hacky-do to make rewrite work
            copy_response_headers {
                include Content-Disposition X-Accel-Redirect
            }

            rewrite * {http.response.header.X-Accel-Redirect}
            file_server {
                root ./api/data
            }
        }
    }
    reverse_proxy /.well-known/* http://localhost:3001
    reverse_proxy /notes/* http://localhost:3001
    log {
        output stdout
        format console
    }
    encode gzip zstd
}

Originally Documented Behavior

The original PR showed this example:

reverse_proxy localhost:8080 {
	@accel header X-Accel-Redirect *
	handle_response @accel {
		root * /path/to/private/files
		rewrite {http.response.header.X-Accel-Redirect}
		file_server
	}
}

That no longer works.

References

Keywords: X-Accel-Redirect, X-Sendfile, X-LIGHTTPD-send-file

Ref:

• https://github.com/caddyserver/caddy/issues/3828
• https://github.com/caddyserver/caddy/pull/4021
• https://github.com/caddyserver/caddy/issues/5208#issuecomment-1954933375