buildkite
Buildkite logs redactor patterns are evaluated greedily, and short values still redacted
TL;DR
The log redactor redacts vars named FOO_SECRETS (plural), as well as short values (< 6 bytes), while iiuc it's not supposed to do either.
Buildkite Agent version: 3.113.0 — though it seems the happen in earlier versions too (e.g. 3.107.0)
Expected Behavior
I'd expect that the log redactor would follow the same behavior as pipeline upload --reject-secrets and only redact values of env vars that match the BUILDKITE_REDACTED_VARS patterns exactly, anchoring them when evaluating them against the env var names.
That means that BUILDKITE_REDACTED_VARS=*_SECRET should redact the value of echo $FAKE_SECRET but not the value or echo $FAKE_SECRETS.
I'd expect that small values like 1 and true would not be redacted.
Actual Behavior
The log redactor considers env vars named $FOO_SECRETS as being secret, while the *_SECRET pattern (from the default value of --redacted-vars / $BUILDKITE_REDACTED_VARS) should match $FOO_SECRET but not $FOO_SECRETS.
💭 It seems to me that when it applies the patterns from BUILDKITE_REDACTED_VARS on the env var names—to know which one to consider needing redaction—it doesn't anchor the pattern on the env var name being tested?
As a result, even with BUILDKITE_REDACTED_VARS's default value of *_PASSWORD,*_SECRET,*_TOKEN,*_PRIVATE_KEY,*_ACCESS_KEY,*_SECRET_KEY,*_CONNECTION_STRING, values of env vars with names like MY_SECRETS or DISABLE_PASSWORD_PROMPT will be matched and redacted in the logs.
Also, env vars considered secrets but having a value like 1 or true are also redacted in the logs, despite those values being shorter than 6 bytes.
How I found out
See https://github.com/buildkite/agent/pull/3580#issuecomment-3554186310
I recently updated to buildkite-agent version 3.113.0 to benefit from the recent fix about buildkite-agent pipeline upload --reject-secrets.
In order to make this new behavior be automatically enabled for all our call to buildkite-agent pipeline upload in all our pipelines, I've set BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=1 in the env hook[^1]. But that had the side effect of redacting all the characters 1 in my logs 😱
[2025-11-19T18:44:29Z] 2025-[REDACTED][REDACTED]-[REDACTED]9 [REDACTED]8:44:29 INFO Reading pipeline configs from [".buildkite/pipeline.yml"]
[2025-11-19T18:44:29Z] 2025-[REDACTED][REDACTED]-[REDACTED]9 [REDACTED]8:44:29 INFO Updating BUILDKITE_COMMIT to "2fa[REDACTED]2d2c[REDACTED]da[REDACTED][REDACTED]3534ab8aa5a89d3bf5b4f376[REDACTED]cd"
[2025-11-19T18:44:30Z] 2025-[REDACTED][REDACTED]-[REDACTED]9 [REDACTED]8:44:30 INFO Successfully parsed and uploaded pipeline #[REDACTED] from "pipeline.yml"
[^1]: In practice I did that in our s3://ci-secrets/env file in S3 that is then injected in all our pipelines via Buildkite's s3secrets-helper
Switching the value to BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true stopped all the 1 in my logs from being redacted, so that's better…
…But now if I have the word true anywhere in my log, that one will still be redacted 😒
echo "Please do not censor true statements! The truth must be told!"
[2025-11-19T21:08:31Z] Please do not censor [REDACTED] statements! The truth must be told!
Happening in earlier versions too
While I only discovered this while updating to 3.113.0, when I looked at jobs running on some of our other queues that are using an older CloudFormations stack and buildkite-agent version I still saw some odd redactions, like here on an agent running 3.107.0:
Discrepancy with buildkite-agent pipeline upload --reject-secrets
Interestingly, the --reject-secrets flag of pipeline upload isn't subject to that bug. For example, with a pipeline containing echo $FAKE_SECRETS, it will not reject the pipeline upload (unless if we explicitly override the patterns to contain the plural form too, i.e. BUILDKITE_REDACTED_VARS='*_SECRETS').
Test example
Test pipeline:
# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
---
steps:
- label: "Test secrets leakage"
command: |
echo "FAKE_SECRETS: $FAKE_SECRETS"
Running with default redacted vars patterns:
$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
-e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true \
buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 19:58:40 INFO Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
steps:
- label: Test secrets leakage
command: |
echo "FAKE_SECRETS: this-must-not-leak"
Running with custom redaction patterns but still without plural form:
$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
-e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true -e BUILDKITE_REDACTED_VARS='*_SECRET' \
buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 20:00:34 INFO Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
steps:
- label: Test secrets leakage
command: |
echo "FAKE_SECRETS: this-must-not-leak"
Running with custom redaction patterns using the plural form:
$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
-e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true -e BUILDKITE_REDACTED_VARS='*_SECRETS' \
buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 20:00:40 INFO Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
2025-11-19 20:00:40 WARN Some variables have values below minimum length (6 bytes) and will not be redacted: BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS
buildkite-agent: fatal: pipeline "pipeline.yml" contains values interpolated from the following secret environment variables: [FAKE_SECRETS], and cannot be uploaded to Buildkite
Custom redaction patterns allowing any suffix:
$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
-e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true -e BUILDKITE_REDACTED_VARS='*_SECRET*' \
buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 21:17:34 INFO Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
2025-11-19 21:17:34 WARN Some variables have values below minimum length (6 bytes) and will not be redacted: BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS
buildkite-agent: fatal: pipeline "pipeline.yml" contains values interpolated from the following secret environment variables: [FAKE_SECRETS], and cannot be uploaded to Buildkite
One thing I noted is that when I run the pipeline upload agent locally via Docker like in the code snippets above, I get a WARN Some variables have values below minimum length (6 bytes) and will not be redacted warning in stdout. Which is great and what I want indeed!
But when I look at the logs of a build that runs on Buildkite directly running on our EC2 instances with the same 3.113.0 agent version, I don't see such warning:
Which I guess is consistent with the fact that my true is [REDACTED] at that point in the job logs on our EC2 instances, where I didn't get the warning.
But I'm failing to see what's the difference between running the pipeline upload command from docker run buildkite/agent:3.113.0 locally vs it running on EC2 with same agent version 3.113.0… 🤔
Hi @AliSoftware, thanks for raising this issue.
To start with, I tried replicating the issue with the agent just running on my laptop. With vars set in an agent environment hook, it correctly redacted only the environment variables that strictly matched the --redacted-vars patterns and were at least 6 characters long. So there's some difference between my laptop and your EC2 instances.
Both the naming issue and the length issue are symptomatic of the secret being added to the redactors without being provided by redact.Vars, since that function applies both checks.
There are three ways that a value can bypass redact.Vars and be added to the redactors directly:
BUILDKITE_AGENT_JOB_API_TOKENis added directly. This doesn't seem relevant, it's named correctly, and is a lengthy token generated internally by the agent.- After fetching a secret from the secrets service.
BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETSdoesn't seem like a value you'd be setting with Buildkite Secrets. - When the local Job API is told to redact a value. This is the API used by the
buildkite-agent redactor addcommand.
Based on your job logs, I don't think you are manually calling buildkite-agent redactor add with env vars. Rather, you're using s3-secrets-hooks which is what is calling it.
Reviewing the implementation there, I see that it is filtering the common secret "suffixes" with strings.Contains instead of strings.HasSuffix. It also applies no minimum length check. This neatly explains why this is happening with BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS (with either 1 or true).
It seems reasonable that there is no name check or minimum length check in buildkite-agent redactor add or the Job API - if a customer wants to redact a string badly enough to call buildkite-agent redactor add, then so be it. But this is also an assumption we could revisit.
Based on your job logs, I don't think you are manually calling buildkite-agent redactor add with env vars. Rather, you're using s3-secrets-hooks which is what is calling it.
I can confirm this indeed.
- We don't use the Buildkite Secrets feature
- We don't call
buildkite-agent refactor addmanually either - Instead the
BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETSenv var is indeed set vias3-secrets-hooks, by having it exported in thes3://a8c-ci-secrets/envfile at the root of our bucket (so that it is applied to all our pipelines)
It seems reasonable that there is no name check or minimum length check in buildkite-agent redactor add or the Job API - if a customer wants to redact a string badly enough to call buildkite-agent redactor add, then so be it. But this is also an assumption we could revisit.
Maybe s3-secret-helper could instead call redactor add with a new dedicated --apply-length-check --apply-patterns here, which would be flags you add to redactor add to allow opt in of those checks?
That way the behavior of people explicitly calling redactor add manually (to intentionally redact a specific secret regardless of the patterns or length check) would be unchanged (backwards compatibility/ no breaking change), while when it's called by other tools like s3-secrets-helper on an input not provided explicitly by a user but derived from a list built dynamically instead (where we might not want to make the same assumptions) it can apply the conditions?
Btw FYI, some more context to our specific use case to explain why we set that BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS env var via s3-secrets-helper even if that variable value isn't really a secret value per se, in case you have suggestions for other ways to do it:
- We have multiple agents in our cluster, some being EC2 instances using the Buildkite CF stack and its default Buildkite AMI, some being EC2 instances with custom AMIs where we installed some additional tools, some being self-hosted MacMinis in our DataCenters, some being Docker containers running in k8s. All of them have S3Secrets configured.
- We declare all our pipelines via Terraform (
resource "buildkite-pipeline"which set the samesteps = templatefile(…)for all of those pipelines so that we use the same script doingsource .buildkite/shared-pipeline-varsthenbuildkite-agent pipeline upload "${PIPELINE:-pipeline.yml}"on all of them.- So at first I thought I'd add
--reject-secretsto our template used in terraform so that all pipelines adopt that change. - But we also have some repos that add steps to their current pipeline dynamically by calling
buildkite-agent pipeline upload publish.ymland similar inside their ownpipeline.yml, and I want to enforce--reject-secretson those cases too - That is why I figured setting the env var globally at the agent level would be more fitting to enforce this everywhere.
- So at first I thought I'd add
- But while adding a custom
environmentagent hook would be easy enough on our self-hosted MacMinis or k8s pods or even on agents backed by a CF stack using custom AMI… it is not as trivial for our EC2 instances that use the Buildkite stock AMI (like ourdefaultanduploadqueues) and not a customImageId.
It would have felt overkill to set up all the packer code to create a custom AMI just to add an agent environment hook to then use that AMI as custom ImageId for our default and upload CF stacks.
That's is why declaring that env var via the env file sourced by s3-secrets-helper thus seemed to me like the easiest way to have that env var guaranteed to be declared in all our agents, without having to create custom AMIs just for that or having to that change in every single custom agent we have just for that.
If you have suggestions on an other way (not relying on s3-secrets-helper) how to declare that env var on all our agents easily and without having to resort to custom AMIs, I'm interested, as iiuc that would solve the extraneous redaction issue if it's not set as a S3 secret to begin with 🤞
@DrJosh9000 I took a stab at trying to fix this in https://github.com/buildkite/elastic-ci-stack-s3-secrets-hooks/pull/146 (Bear with me, this is my first time writing Go 😅 but I figured it was a good occasion for me to get my feet wet in golang)