agent-stack-k8s icon indicating copy to clipboard operation
agent-stack-k8s copied to clipboard

[WIP] Pre-schedule hook

Open DrJosh9000 opened this issue 9 months ago • 0 comments

WIP: This currently doesn't work. The controller container simply doesn't have permissions to execute a script. (That sounds sensible actually?) So it will have to go in another container.


The stack can configure the agent to run hooks easily. In fact the agent container ships with a config file that sets the hook path to /buildkite/hooks, so one merely need mount some hooks at that path and the agent will run them, including pre-bootstrap.

However, with podSpecPatch, a pipeline can trivially override the hooks path (e.g. setting a BUILDKITE_HOOKS_PATH env var) which would disable pre-bootstrap being run. So pre-bootstrap no longer works as a mechanism to check jobs before running.

This PR adds a new special hook for the k8s controller: pre-schedule. Similar to pre-bootstrap it has the ability to inspect the job the controller is about to schedule a pod for. Rather than "env var" form, I've opted to dump the whole job as JSON.

Since this happens in the scheduler, I've gone ahead with a long-desired refactor of some of the scheduler methods to eliminate the err struct field.

DrJosh9000 avatar May 01 '24 04:05 DrJosh9000