vault
vault copied to clipboard
hashicorp
KV path with many segments errors with MySQL column size
Description
Environment
Vault v1.3.4
Linux XXXXXXXXXXXX 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
mysql Ver 14.14 Distrib 5.7.34, for Linux (x86_64) using EditLine wrapper
Using a path in the KV store with many path segments will raise a MySQL error:
Error 1406: Data too long for column 'vault_key' at row 1
The length of the KV path is not the problem, it specifically triggers on the number of KV path segments.
There is nothing written about this limitation in the documentation https://www.vaultproject.io/docs/internals/limits and https://www.vaultproject.io/docs/configuration/storage/mysql.
Ref issue https://github.com/hashicorp/vault/issues/11016 which identifies the same problem.
Reproduction
Configuration
storage "mysql" {
database = "vault"
table = "vault"
username = "vault"
password = "vault"
}
Start Vault
vault server -dev -config ./vault.hcl
The table in MySQL is created.
$ mysql -uroot vault
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 267
Server version: 5.7.34 MySQL Community Server (GPL)
Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show tables;
+-----------------+
| Tables_in_vault |
+-----------------+
| vault |
+-----------------+
1 row in set (0.00 sec)
mysql> desc vault;
+-------------+----------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+----------------+------+-----+---------+-------+
| vault_key | varbinary(512) | NO | PRI | NULL | |
| vault_value | mediumblob | YES | | NULL | |
+-------------+----------------+------+-----+---------+-------+
2 rows in set (0.00 sec)
Setting key with many path segments fails, but setting very long key succeeds
$ vault kv put secret/a/a/a/a/a/a/a/a/a a=b
Error writing data to secret/data/a/a/a/a/a/a/a/a/a: Error making API request.
URL: PUT http://127.0.0.1:8200/v1/secret/data/a/a/a/a/a/a/a/a/a
Code: 500. Errors:
* 1 error occurred:
* Error 1406: Data too long for column 'vault_key' at row 1
$ vault kv put secret/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa a=b
Key Value
--- -----
created_time 2021-05-27T10:59:55.179724556Z
deletion_time n/a
destroyed false
version 1
Vault v1.3.4
Linux XXXXXXXXXXXX 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
mysql Ver 14.14 Distrib 5.7.34, for Linux (x86_64) using EditLine wrapper
I can also reproduce this with a recent version of both Vault and MySQL:
Vault v1.7.2 (db0e4245d5119b5929e611ea4d9bf66e47f3f208)
mysql Ver 8.0.22-13 for Linux on x86_64 (Percona Server (GPL), Release '13', Revision '6f7822f')
I see the same thing with vault 1.8.1, getting error on kv2 trying to add a path: secret/apkjflkjdslkfjlkdsjflkdsjflkdsjflkdsjlkdsjkfjlkdsjf/kljhkjdsfakjdshkjdshfkjhdskjfhdskjhfkdjsfh/kjhbkjfhkjdshfkjhdskjshfku
on kv1 this works fine.
Same error. vault 1.3.0, path secret/booo/booooooooooooo.boo/booooooooo/boo/booooBooooBoooBo/booooo (boos are replacements for real letters). Altered table to have vault.vault_key varbinary(1024).
My org hit this issue as well. We're currently running vault 1.9.3 from the hashicorp debian repos, mariadb 10.3.31-MariaDB-0+deb10u1-log from the debian buster repos, and have KVv2 mounted at kv.
Experimentally, the put seems to start failing once 8 user-defined path segments are present:
$ vault kv put kv/a/b/c/d/e/f/g/h a=b
Error writing data to kv/data/a/b/c/d/e/f/g/h: Error making API request.
URL: PUT <snip>/v1/kv/data/a/b/c/d/e/f/g/h
Code: 500. Errors:
* 1 error occurred:
* Error 1406: Data too long for column 'vault_key' at row 1
As a workaround, we've doubled the size of the vault_key column as recommended by @isbear:
ALTER TABLE `vault` CHANGE `vault_key` `vault_key` VARBINARY(1024) NOT NULL;
This starts failing once 17 user-defined path segments are present. This limit is less likely to be hit, but doesn't actually solve the underlying issue.
$ vault kv put kv/a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q a=b
Error writing data to kv/data/a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q: Error making API request.
URL: PUT https://vault-dev.cs.uni-paderborn.de:8200/v1/kv/data/a/b/c/d/e/f/g/h/i/j/k/l/m/n/o/p/q
Code: 500. Errors:
* 1 error occurred:
* Error 1406: Data too long for column 'vault_key' at row 1
We have also hit this issue, but get a different error. I'm using Vault server v1.9.3 and Vault cli v1.10.0.
We see no errors logged on the creation:
vault kv put secrets/tribe/sandbox/alice/alice/app/key/for/tags foo=bar
================== Secret Path ==================
secrets/data/tribe/sandbox/alice/alice/app/key/for/tags
======= Metadata =======
Key Value
--- -----
created_time 2022-04-04T08:55:05.521219269Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1
But attempting to list subpaths does fail:
vault kv list secrets/tribe/sandbox/alice/alice/app/key
Error listing secrets/metadata/tribe/sandbox/alice/alice/app/key: Error making API request.
URL: GET https://platform/v1/secrets/metadata/tribe/sandbox/alice/alice/app/key?list=true
Code: 400. Errors:
* 1 error occurred:
* invalid ciphertext: wrong number of fields
This is the same as #6805 which was closed in 2019, and is also the same as this discussion from March 2020.
I can confirm that increasing the field size to 1024 does allow for a larger number of segements in the path, but this apparently needs innodb_large_prefix set in order to go beyond 767 bytes. This might warrant further investigation.
Seems like this is finally solved: https://github.com/hashicorp/vault/pull/14231
I can confirm this issue is fixed in Vault 1.13
vault --version
Vault v1.13.0 (a4cf0dc4437de35fce4860857b64569d092a9b5a), built 2023-03-01T14:58:13Z
Write a KV secret with many segments.
vault kv put secret/a/a/a/a/a/a/a/a/a a=b
======== Secret Path ========
secret/data/a/a/a/a/a/a/a/a/a
======= Metadata =======
Key Value
--- -----
created_time 2023-03-20T09:59:20.667231079Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1