Dynamically set redacted vars

[ChrisBr]

As part of our bootstrap process we decrypt ejson files. It would be great if we could dynamically update the redacted vars after we decrypted the ejson files with the keys in it.

Something like

EXISTING_VARS = $(agent redacted-vars get)
agent redacted-vars set "$EXISTING_VARS,FOO,BAR,FOOBAR"

[triarius]

Thanks for raising this @ChrisBr, I think it makes more sense to have agent redacted-vars add instead.

Implementing this feature will require some delicate changes to how the redactor works, which we think are possible but don't plan to flesh out in detail right now. We suggest, if anyone wants to take this up,

1. have a look at modifying the redactor to be updated with new values to redact.
2. Add the ability to update redacted values to the Job API
3. agent redacted-vars add will call the Job API, sourcing the values from its own environment based on the keys provided as arguments.

[triarius]

@ChrisBr I'm about to introduce an agent command that would allow you to add values to the redactor during a running job. I think it will work with your ejson use case. Something like

ejson decrypt foo.ejson | buildkite-agent redactor add --format json

should work if the ejson is flat, though it will also redact the values beginning with _.

If you have nesting in your ejson, then I think it will be relatively straightforward to extend this to add nested values to the redactor too. LMK if you would like me to take a look, though I might do that in a separate PR.

[ChrisBr]

@triarius that's great, thanks.

Redacting _ values shouldn't be a big problem.

We probably have nesting in the ejson files. What's the format the redactor expects? Can we just give it a list of keys in JSON format? If so we can probably just format it ourselves?

[triarius]

What's the format the redactor expects?

Currently, just string valued JSON. So if ejson emits something of the form

{
  "_public_key": "63ccf05a9492e68e12eeb1c705888aebdcc0080af7e594fc402beb24cce9d14f",
  "_database_username": "1234username",
  "database_password": "hunter2"
}

then 63ccf05a9492e68e12eeb1c705888aebdcc0080af7e594fc402beb24cce9d14f, 1234username, and hunter2 will be added to the redactor.

But if the JSON object has arrays, objects, boolean values etc, it won't work.

Can we just give it a list of keys in JSON format? If so we can probably just format it ourselves?

Not as it's currently written. The accepted formats will have limited configurability. One thing you might be able to do once this is merged is to use jq to flatten the structure after ejson decrypts it. For example, if it's like:

{
  "_public_key": "63ccf05a9492e68e12eeb1c705888aebdcc0080af7e594fc402beb24cce9d14f",
  "databases": [
    {
      "siamese": {
        "_username": "1234username",
        "password": "hunter2"
      }
    },
    {
      "persian": {
        "_username": "1234username",
        "password": "hunter3"
      }
    }
  ]
}

Then the jq expression:

jq 'tostream | select(length == 2) | {(.[0] | join(".")): .[1]}' | jq -s add

should flatten it to become:

{
  "_public_key": "63ccf05a9492e68e12eeb1c705888aebdcc0080af7e594fc402beb24cce9d14f",
  "databases.0.siamese._username": "1234username",
  "databases.0.siamese.password": "hunter2",
  "databases.1.persian._username": "1234username",
  "databases.1.persian.password": "hunter3"
}

Piping this into buildkite-agent redactor add --format json should redact all the usernames and passwords (and the _public_key).

[ChrisBr]

One thing you might be able to do once this is merged is to use jq to flatten the structure after ejson decrypts it.

Yup that should work 👍 Thanks for working on this.