vault icon indicating copy to clipboard operation
vault copied to clipboard

PROXY protocol listener with `deny_unauthorized` silently closes after a non-authorized request

Open bmendric opened this issue 2 years ago • 4 comments

Describe the bug A TCP listener configured with proxy_protocol_behavior = "deny_unauthorize" silently closes after receiving a request originating from an IP not listed within the proxy_protocol_authorized_addrs list.

To Reproduce Steps to reproduce the behavior:

  1. Run a vault server with the configuration provided below: vault server -config=config.hcl
  2. Validate that the service is running and bound to the expected port: curl http://127.0.0.1:8200/v1/sys/health / ss -tlpn | grep 8200
  3. Perform a query against the Vault server using an IP address not contained within the proxy_protocol_authorized_addrs list. In my case, curl http://192.168.1.20:8200/v1/sys/health
  4. Validate that the request was not serviced and the listener is no longer active: ss -tlpn | grep 8200

Expected behavior Either:

  1. Anything to log an error saying that the listener closed and why
  2. For the listener to reject the connection and carry on with listening

Environment:

  • Vault Server Version (retrieve with vault status): Vault v1.11.3 (17250b25303c6418c283c95b1d5a9c9f16174fe8), built 2022-08-26T10:27:10Z
  • Vault CLI Version (retrieve with vault version): Vault v1.11.3 (17250b25303c6418c283c95b1d5a9c9f16174fe8), built 2022-08-26T10:27:10Z
  • Server Operating System/Architecture: OpenSUSE Tumbleweed x86_64

Of note, the above environment is my home lab and reflects the simple configuration testing done/described in this issue. The problem was originally discovered in a production testing environment using a three node cluster of Ubuntu 18.04 nodes running Vault 1.11.1 from within Docker containers.

Vault server configuration file(s):

storage "file" {
  path = "/tmp/vault/data"
}

listener "tcp" {
  address                         = "[::]:8200"
  tls_disable                     = true
  proxy_protocol_behavior         = "deny_unauthorized"
  proxy_protocol_authorized_addrs = "127.0.0.1"
}

log_level  = "trace"
log_format = "json"
ui         = false

disable_mlock = true

Additional context This took me quite a long time to wrap my head around due to our configuration in the production testing environment, so there are a couple interesting bits I picked up along the way.

  1. This behavior does not appear with proxy_protocol_behavior set to use_always nor allow_authorized
  2. The packet capture is what I expect (I think?) -- TCP handshake followed by a request and the Vault server sending a reset directly afterwards due to the lack of a PROXY protocol message image

Passing --haproxy-protocol to curl resulted in the following: image

  1. Slightly surprising to me, the cluster port for the HA cluster is unaffected by the API port closure
  2. I think there is a bug with how reject errors are handled in the proxy protocol wrapping function as the pires/go-proxyproto implementation seems to blindly close the TCP connection any time a policy resolution returns an non-nil error. I am fairly confident this isn't related, but thought it interesting

Finally, here is a screenshot of my local testing alongside the runtime logs -- image

image

bmendric avatar Sep 14 '22 23:09 bmendric