Simple options to protect agents in case of Buildkite being compromised

[david-poirier]

Given this scenario - buildkite gets compromised and malicious jobs are sent to some or all agents connected to the buildkite agent controller - I feel that the current (and proposed) mitigations are either insufficient or too complex.

Current advice (at https://buildkite.com/docs/agent/v3/securing#restricting-access-by-the-buildkite-agent-controller) is:

• use hooks to allow-list repos, plugins, and commands
• disable local hooks
• disable plugins
• disable command eval (which also disables plugins)

Using hooks does work but is difficult and error-prone at scale - ensuring they work as intended across different operating systems, and aren't removed, overwritten or broken by other engineers with limited security background is... challenging.

Disabling local hooks and plugins solves parts of the scenario but removes critical and commonly used agent functionality.

Disabling command eval solves a big part of the scenario, but by disabling plugins also removes critical functionality.

Pipeline signing (https://github.com/buildkite/buildkite-signed-pipeline) does work as well, but requires significant engineering effort to implement in its current form (and less but still significant effort once it's integrated into the agent).

---

I'd like to propose a simple set of agent configuration options that I believe can protect against the buildkite gets compromised scenario, without the complexity of hooks or pipeline signing.

• Protection from malicious repositories - An allowed-repositories command line param accepting an allow-list of regular expression strings. e.g.: ^[email protected]/buildkite/.*$. If the param is used and a job arrives with a BUILDKITE_REPO value that doesn't match an entry in the list then the job is rejected - merged PR here.
• Protection from malicious plugins - An allowed-plugins command line param accepting an allow-list of regular expression strings. e.g.: ^github.com/buildkite/.*$. If the param is used and a job arrives with BUILDKITE_PLUGINS containing a plugin that doesn't match an entry in the list then the job is rejected. - merged PR here
• Protection from malicious environment variables - An allowed-environment-variables command line param accepting an allow-list of regular expression strings, e.g.: ^BUILD_.*$. If the param is used then any environment variables that don't match an entry in the list are ignored. - merged PR here
• Protection from malicious commands - A command-mode command line param with possible values of shell, repo-only-executable, or repo-only-executable-no-plugins. If shell is specified then the command is passed to the shell for evaluation, as is normal currently. If repo-only-executable is specified then BUILDKITE_COMMAND is tokenized (using shlex.Split) and the first token must refer to an executable file in the repository referenced in BUILDKITE_REPO, with subsequent tokens being passed as arguments. If repo-only-executable-no-plugins is specified then behaviour is same as if disable-command-eval is specified (suggest deprecating this). - PR here

I'm happy to produce PRs for all of these suggestions to enable further exploration and discussion.

[moskyb]

hey @david-poirier - love the thinking behind this, and happy to review PRs as they come in.

the only one i can see being a little contentious is the command-mode one, as it has a bit of overlap with the existing no-plugins and no-command-eval, but those options are somewhat blunt instruments, and have fairly nebulous crossover with each other - i'm generally in favour of cleaning it up with the proposed command-mode flag, but we'll se how we go.

[david-poirier]

We're pretty happy with 3 out of 4 here :) I'm going to close this, thanks very much!