JSON configurable limits provided configuration is ignored

[giuse-boccia]

Vault 1.20.3 introduced JSON configurable limits to HTTP handling for JSON payloads: max_json_depth, max_json_string_value_length, max_json_object_entry_count, max_json_array_element_count (https://github.com/hashicorp/vault/pull/31069)

The max_json_string_value_length defaults to 1024 * 1024. We configured it to 1024 * 1024 * 32 in our listener configuration. However, despite this, encrypt and decrypt payloads on transit secret engine with JSON strings bigger than 1 MB (but smaller than 32 MB) return error 500: JSON string value exceeds allowed length.

• We tried to pass the parameter both as a string and as a number.
• We left the max_request_size to its default (32 MB)

To Reproduce Steps to reproduce the behavior:

1. Run cat big_file.txt | base64 | jq -Rs '{plaintext: .}' | vault write transit/encrypt/my-key - (where big_file.txt is a 1.2 MB text file)
2. See error JSON string value exceeds allowed length

Expected behavior We expect the error to be returned with JSON payloads bigger than the value provided in the config (32 MB), not the default value (1 MB)

Environment:

• Vault Server Version: 1.20.3
• Vault CLI Version (retrieve with vault version): 1.20.3
• Server Operating System/Architecture: cos-117 / X86_64

Vault server configuration file:

ui = true
cluster_addr  = "https://**********"
api_addr      = "https://***********"

listener "tcp" {
  telemetry {
    unauthenticated_metrics_access = true
  }

  address         = "0.0.0.0:8200"
  cluster_address = "0.0.0.0:8201"
  tls_cert_file   = "/vault/ssl/vault.crt"
  tls_key_file    = "/vault/ssl/vault.key"
  max_json_string_value_length = 33554432 # also "33554432" doesn't work
  max_request_duration = "52s"
  x_forwarded_for_authorized_addrs = "*******************"
  x_forwarded_for_hop_skips = "1"
  x_forwarded_for_reject_not_authorized = "true"
  x_forwarded_for_reject_not_present = "false"
}

telemetry {
  disable_hostname = true
  prometheus_retention_time = "24h"
}

plugin_directory = "/vault/plugins"
storage "postgresql" {
  connection_url = "***************"
  ha_enabled    = "true"
}

seal "gcpckms" {
  project     = "***********"
  region      = "global"
  key_ring    = "**********"
  crypto_key  = "*************"
}

Additional context

• We did not explicitly configure the transit cache
• The listener documentation page does not mention these new parameters.
• For the moment, we are rolling back to version 1.20.2 to mitigate this.

[heatherezell]

Hello! I'm not sure if you're an Enterprise customer as well - some transit features are Enterprise features as well. If you are, I recommend opening a ticket with support to get the fastest response time. Please feel free to share this issue ID with them as well. Thanks! :)

[giuse-boccia]

Hello, thank you for your response.

I can confirm that this is a bug introduced in Vault 1.20.3. Rolling back to 1.20.2 solves this.

For the moment, we'll stick with 1.20.2 until this bug is fixed. Let me know if you need additional info (beyond the config above) to replicate the issue :-)

[davidadeleon]

Hi @giuse-boccia! Thanks for submitting this issue. I've been unable to replicate this on my end using 1.20.3 as mentioned. Using the max_json_string_value_length provided in your description and using a payload of 1.2mb I was able to successfully encrypt and decrypt using transit. One thing I would quickly validate is Vault's running config. You can run a vault read sys/config/state/sanitized and look for the listener config to make sure that the config option was parsed and applied correctly. Also check the Vault logs to see if there was an error emitted during startup around parsing the config.

[blo0dskAr]

hello, currently we are facing similar issues after upgrading to 1.20.3

reproducing the issue locally on a dev server with untouched default configuration:

• base64 -w0 < 512kB.txt | vault write transit/encrypt/<myKey> plaintext=- ✅ works without issues
• base64 -w0 < 1MB.txt | vault write transit/encrypt/<myKey> plaintext=- 🔴 errors with exceeding length

starting the dev-server with max_json_string_value_length set to 4194304 (via passed -config=<pathToConfig.hcl)

• base64 -w0 < 1MB.txt | vault write transit/encrypt/<myKey> plaintext=- ✅ works without issues

After applying the configuration on the our 3node-production-cluster & restarted the service on all nodes, reading sys/config/state/sanitized now shows the max_json_string_value_length is set correctly. Testing base64 -w0 < 1MB.txt | vault write transit/encrypt/<myKey> plaintext=- Still doesn't work most of the Times. For unclear reason its flaky behavior between working and non-working...

All the Server showed the max_json_string_value_length correctly in the listener in the startup logs when showing Vault Server Configuration.

[jskilton]

Can also reproduce this issue. It is NOT on every call but constant... i have a job and 1 will go through, one will fail, one will go through, one will fail, not every other one, but constantly flaky for the transit encryption engine.

Version 1.20.2 - NO ISSUES Version 1.20.3 and 1.20.4 - issues all the time.

org.springframework.vault.VaultException: Status 500 Internal Server Error: JSON string value exceeds allowed length

no matter what values you put for the new params, and confirmed that the running config did TAKE.

[davidadeleon]

Hi all! We were able to reproduce the issue after the additional information provided by @blo0dskAr and are looking into a resolution. A workaround for now would be to set the X-Vault-No-Request-Forwarding header for each request, or target the active node in the cluster specifically. Unfortunately this issue isn't exactly straight forward to address and is going to require a bit of time to resolve, but we are aware and have gotten this issue in front of the appropriate team.

[whetchar]

Hi! The issue is still present in 1.21.1. Any expected date to release a fix?

[jskilton]

Still broken in 1.21.1 ... we really want to upgrade. Can we get a timeline or where this is in the list?

[heatherezell]

Hi friends! We are putting the final touches on this fix in preparation for it being included in the next release. I really appreciate everyone's patience!

[rathberm]

@heatherezell It seems like this was backported as a security fix to v1.19 enterprise only releases, e.g 1.19.11. Can you confirm this? I simultaneously opened support ticket 203756 as an enterprise customer.