Add support for storing secrets in meta-data

[testworksau]

Build meta-data cannot be seen via the Buildkite UI, but - curiously - can be returned via the API. This gives a false sense of security to those who think using meta-data is a secure way of passing secret data around.

It would be beneficial to be able to call:

buildkite-agent secret-meta-data set

or

buildkite-agent meta-data set <key> <value> --secret

or similar, and have that metadata stay hidden from the API.

Use case

In the example below, pipeline step 1 operates on an agent queue that has access to secrets, and pipeline step 2 does not.

1. Pipeline step 1 retrieves a secret, and adds the secret as meta-data so that it can be used by the next step in the build
2. Pipeline step 2 reads the secret and does xyz with it

Notes

There is nothing I can find in the documentation that suggests secrets shouldn't be passed around via meta-data. Is there any guidance on a recommended approach?

https://buildkite.com/docs/agent/v2/cli-meta-data

[keithduncan]

In lieu of an agent feature that handles this transparently, it would be possible to encrypt / decrypt meta-data values using something like AWS KMS, granting asymmetric encrypt / decrypt permission to your different agent pools.

I’ve changed the issue title to suggest adding support for secret values in the meta-data store rather than a flag that hides them from some API responses but returns the clear-text to a get operation.