vault
vault copied to clipboard
PROXY protocol listener with `deny_unauthorized` silently closes after a non-authorized request
Describe the bug
A TCP listener configured with proxy_protocol_behavior = "deny_unauthorize"
silently closes after receiving a request originating from an IP not listed within the proxy_protocol_authorized_addrs
list.
To Reproduce Steps to reproduce the behavior:
- Run a vault server with the configuration provided below:
vault server -config=config.hcl
- Validate that the service is running and bound to the expected port:
curl http://127.0.0.1:8200/v1/sys/health
/ss -tlpn | grep 8200
- Perform a query against the Vault server using an IP address not contained within the
proxy_protocol_authorized_addrs
list. In my case,curl http://192.168.1.20:8200/v1/sys/health
- Validate that the request was not serviced and the listener is no longer active:
ss -tlpn | grep 8200
Expected behavior Either:
- Anything to log an error saying that the listener closed and why
- For the listener to reject the connection and carry on with listening
Environment:
- Vault Server Version (retrieve with
vault status
):Vault v1.11.3 (17250b25303c6418c283c95b1d5a9c9f16174fe8), built 2022-08-26T10:27:10Z
- Vault CLI Version (retrieve with
vault version
):Vault v1.11.3 (17250b25303c6418c283c95b1d5a9c9f16174fe8), built 2022-08-26T10:27:10Z
- Server Operating System/Architecture: OpenSUSE Tumbleweed x86_64
Of note, the above environment is my home lab and reflects the simple configuration testing done/described in this issue. The problem was originally discovered in a production testing environment using a three node cluster of Ubuntu 18.04 nodes running Vault 1.11.1 from within Docker containers.
Vault server configuration file(s):
storage "file" {
path = "/tmp/vault/data"
}
listener "tcp" {
address = "[::]:8200"
tls_disable = true
proxy_protocol_behavior = "deny_unauthorized"
proxy_protocol_authorized_addrs = "127.0.0.1"
}
log_level = "trace"
log_format = "json"
ui = false
disable_mlock = true
Additional context This took me quite a long time to wrap my head around due to our configuration in the production testing environment, so there are a couple interesting bits I picked up along the way.
- This behavior does not appear with
proxy_protocol_behavior
set touse_always
norallow_authorized
- The packet capture is what I expect (I think?) -- TCP handshake followed by a request and the Vault server sending a reset directly afterwards due to the lack of a PROXY protocol message
Passing --haproxy-protocol
to curl resulted in the following:
- Slightly surprising to me, the cluster port for the HA cluster is unaffected by the API port closure
- I think there is a bug with how reject errors are handled in the proxy protocol wrapping function as the pires/go-proxyproto implementation seems to blindly close the TCP connection any time a policy resolution returns an non-nil error. I am fairly confident this isn't related, but thought it interesting
Finally, here is a screenshot of my local testing alongside the runtime logs --