LeSuisse
Signing large files.
Hello, I wish there is a discuss tab on github. This is not an issue perse but a question - Is there any way to sign large files with this plugin ? If not, please can you recommend a way to do it ?
I am trying to submit a large json file with base64 encoded data which needs to be signed (300MB) using curl and I get "* failed to parse JSON input: http: request body too large".
Hello,
Thank you for your interest.
So, to be honest I was not really aware of any kind of limitation until you opened this issue. It's not something that I encounter in my daily use cases.
I have been able to reproduce so I dug a bit more to understand what the root cause is.
Vault itself sets a limit to 32MB per request to prevent denial of service attacks: https://github.com/hashicorp/vault/blob/d4f17b8f869e5d0dfacd3baf9bfedd25a69fcdf2/http/handler.go#L52
This limit impacts everything in Vault (e.g. the transit backend) and can not be chosen (or I missed something in the code or in the documentation).
I'm a bit curious how others deal with it when they use the transit backend so I'm gonna start a discussion in the Vault mailing list by the end of the week.
just FYI... even if you bump the limit up and recompile vault, you will crash the vault on eeeeh ~ 150MB files with error fatal error: runtime: out of memory
bottom line, dont waste your time trying to do it :-)
At least with Vault 1.4, you can set max_request_size = 0 in your listener config, and it seems to work; I've used it to sign RPMs that were 250M (338M base64-encoded).
@LeSuisse
I have been trying to get the /gpg/sign endpoint to sign large files (550MB). When I base64 encode and pass my rpm to the endpoint, the upload finishes but then the plugin crashes before sending me the response.
NOTE: I have confirmed my setup works for smaller rpms.
Curl post
> POST /v1/gpg/sign/rpm-signing-key HTTP/1.1
> Host: <vault_instance>:8200
> User-Agent: curl/7.80.0
> Accept: */*
> X-Vault-Token: <token>
> Content-Length: 572175616
> Expect: 100-continue
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 100 Continue
} [5 bytes data]
99 545M 0 0 99 545M 0 1226k 0:07:35 0:07:35 --:--:-- 1565k* We are completely uploaded and fine
100 545M 0 0 100 545M 0 1172k 0:07:56 0:07:56 --:--:-- 0{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< Cache-Control: no-store
< Content-Type: application/json
< Date: Wed, 06 Apr 2022 20:16:25 GMT
< Content-Length: 57
<
{ [57 bytes data]
100 545M 0 57 100 545M 0 1170k 0:07:57 0:07:57 --:--:-- 15
* Connection #0 to host <vault_instance> left intact
{"errors":["1 error occurred:\n\t* internal error\n\n"]}
vault logs
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.321-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: plugin process exited: path=/data/etc/vault/plugin/vault-gpg-plugin pid=21709 error="signal: killed"
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.321-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738: reloading plugin backend: plugin=vault-gpg-plugin
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.322-0500 [WARN] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: error closing client during Kill: err="rpc error: code = Canceled desc = grpc: the client connection is closing"
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.322-0500 [WARN] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: plugin failed to exit gracefully
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.342-0500 [DEBUG] core.cluster-listener: performing server cert lookup
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.342-0500 [DEBUG] core.cluster-listener: error handshaking cluster connection: error="tls: no certificates configured"
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.411-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.522-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: starting plugin: path=/data/etc/vault/plugin/vault-gpg-plugin args=[/data/etc/vault/plugin/vault-gpg-plugin]
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.522-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: plugin started: path=/data/etc/vault/plugin/vault-gpg-plugin pid=24429
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.522-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: waiting for RPC address: path=/data/etc/vault/plugin/vault-gpg-plugin
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.613-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.669-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: using plugin: version=4
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.669-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin.vault-gpg-plugin: plugin address: address=/tmp/plugin2819653601 network=unix timestamp=2022-04-06T15:16:16.669-0500
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.692-0500 [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/h8abb630863340088962df1ad37bb875f3f3c31a7bbbfd0efca2959042bfc8f3b
Apr 06 15:16:16 <vault_instance> vault[1398]: 2022-04-06T15:16:16.815-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:17 <vault_instance> vault[1398]: 2022-04-06T15:16:17.012-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:17 <vault_instance> vault[1398]: 2022-04-06T15:16:17.212-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:17 <vault_instance> vault[1398]: 2022-04-06T15:16:17.412-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:17 <vault_instance> vault[1398]: 2022-04-06T15:16:17.629-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:20 <vault_instance> vault[1398]: 2022-04-06T15:16:20.244-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: plugin process exited: path=/data/etc/vault/plugin/vault-gpg-plugin pid=24429 error="signal: killed"
Apr 06 15:16:20 <vault_instance> vault[1398]: 2022-04-06T15:16:20.244-0500 [ERROR] core: failed to run existence check: error="plugin is shut down"
Apr 06 15:16:22 <vault_instance> vault[1398]: 2022-04-06T15:16:22.929-0500 [DEBUG] core.cluster-listener: performing server cert lookup
Apr 06 15:16:22 <vault_instance> vault[1398]: 2022-04-06T15:16:22.929-0500 [DEBUG] core.cluster-listener: error handshaking cluster connection: error="tls: no certificates configured"
Apr 06 15:16:22 <vault_instance> vault[1398]: 2022-04-06T15:16:22.938-0500 [DEBUG] replication.index.perf: saved checkpoint: num_dirty=0
Apr 06 15:16:22 <vault_instance> vault[1398]: 2022-04-06T15:16:22.941-0500 [DEBUG] replication.index.local: saved checkpoint: num_dirty=0
Apr 06 15:16:22 <vault_instance> vault[1398]: 2022-04-06T15:16:22.941-0500 [DEBUG] replication.index.periodic: starting WAL GC: from=193616 to=193628 last=193884
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.006-0500 [INFO] expiration: revoked lease: lease_id=auth/token/create/hb92bafe0d100fd02b2c7ff379d201a23177954f92de7f397cd46de820bbfe490
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.015-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.211-0500 [DEBUG] replication.index.perf: saved checkpoint: num_dirty=0
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.220-0500 [DEBUG] replication.index.local: saved checkpoint: num_dirty=3
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.220-0500 [DEBUG] replication.index.periodic: starting WAL GC: from=193629 to=193629 last=193885
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.236-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.412-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.611-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.633-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738: reloading plugin backend: plugin=vault-gpg-plugin
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.633-0500 [WARN] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: error closing client during Kill: err="rpc error: code = Canceled desc = grpc: the client connection is closing"
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.633-0500 [WARN] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: plugin failed to exit gracefully
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.812-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.920-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: starting plugin: path=/data/etc/vault/plugin/vault-gpg-plugin args=[/data/etc/vault/plugin/vault-gpg-plugin]
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.920-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: plugin started: path=/data/etc/vault/plugin/vault-gpg-plugin pid=24446
Apr 06 15:16:26 <vault_instance> vault[1398]: 2022-04-06T15:16:26.920-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: waiting for RPC address: path=/data/etc/vault/plugin/vault-gpg-plugin
Apr 06 15:16:27 <vault_instance> vault[1398]: 2022-04-06T15:16:27.012-0500 [DEBUG] replication.index.local: flushed dirty pages: num_pages=1
Apr 06 15:16:27 <vault_instance> vault[1398]: 2022-04-06T15:16:27.037-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: using plugin: version=4
Apr 06 15:16:27 <vault_instance> vault[1398]: 2022-04-06T15:16:27.038-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin.vault-gpg-plugin: plugin address: address=/tmp/plugin964201703 network=unix timestamp=2022-04-06T15:16:27.037-0500
Apr 06 15:16:27 <vault_instance> vault[1398]: 2022-04-06T15:16:27.054-0500 [INFO] expiration: revoked lease: lease_id=sys/wrapping/wrap/h31eb2a75ec0e372f72012f0fe51fb332bc646052ab3d90536093dcb51951d11a
Looks like it was killed:
Apr 06 15:16:20 <vault_instance> vault[1398]: 2022-04-06T15:16:20.244-0500 [DEBUG] secrets.vault-gpg-plugin.vault-gpg-plugin_7c5e4738.vault-gpg-plugin: plugin process exited: path=/data/etc/vault/plugin/vault-gpg-plugin pid=24429 error="signal: killed"
I bet you ran out of memory - I noticed that signing very large blobs bloats the memory footprint rather significantly, and I had to have more memory added to the machine I use to sign stuff.
Yep the memory usage is a known thing and I'm not sure there is a lot we can do since the last time I checked #9
Closing as there is not a lot of things we can do here. The content that needs to be signed must be sent to Vault and signing it consumes memory.