buildkite
Pipeline/build prioritization
It would be great to be able to assign all build jobs from a particular pipeline a higher priority than jobs from another pipeline.
Example use case: A 2 pipeline configuration: one pipeline builds commits on the master branch and is triggered automatically via github events, a second pipeline is used to build and deploy the app and is triggered manually (relatively infrequently). Jobs from both pipelines are runnable on all agents in a shared pool.
We have a setup like this and find that when it comes time to deploy, the jobs in the deploy pipeline often have to wait a very long time for an available agent in the pool (all busy most of the time building things in 'build master" pipeline). Even when busy agents finish their current job, they will often choose another job from the "build master" pipeline, rather than "deploy". This can create long lag times for a manual job we want to run asap. What we want to be able to do is configure any scheduled jobs from the deploy pipeline to always run before any scheduled jobs from the "build master" pipeline. Alternatively, a way to assign a priority to a build (e.g. an option on the create build API), such that jobs from that build are assigned ahead of other scheduled jobs with a lower, or absent, priority.
Thanks for the feedback!
Can I ask if you’ve looked at using build queues to achieve this? If you want to keep agents clear to do timely/urgent work (such as deploys) normally you'd use queues to achieve it: https://buildkite.com/docs/agent/queues
Build queues would definitely work. The downside is that for this setup, as I understand it, we'd need to have at least one agent running tagged as an agent that runs only jobs from the deploy queue. If deploy jobs run only once per day, then having that agent sitting there doing nothing for the rest of the day is a bit of a waste.
We have a wide inventory of jobs, too - some are high CPU/memory/IO (compile etc), some are basically nothing but waiting for some world simulation to finish (I love where I work) and tell us whether there was an apocalypse or something. For the latter, I was thinking to have a tag to indicate this is the node for long-running-wait tasks compared to the former. Deploys are similar.
Like one of the above commenters mentioned, is there any way to implement the following scenario:
I have two queues, one is the deploy queue, which I expect to be empty most of the time, and the default queue, which should be full. Is there any way with what Buildkite already provides, to implement it such that the agents always check for something from the deploy queue first?
Adding another vote for this feature. Building the "X" branch pipeline queue as a priority over every other pipeline queue would be amazing! Running multiple agents has some concurrency issues in this specific case so we can't simply run a dedicated "x" branch agent.
Another vote for this feature! Build queues are insufficient on their own, because they essentially partition scarce resources. We would ideally allow our postmerge jobs to run on all agents at a lower priority, and have Pull Request builds jump the queue and run ASAP, since developers are waiting on those. Even if we allocate 50% of agents to a "postmerge" queue, it still reduces our capacity by 50% (which is bad for developers), and we'd want PR builds to jump the queue on those separate agents in addition to the default queue.
It would be really useful to be able to prioritize queues; In my use-case, an agent generally should belong to a set of "primary" queues, where it pulls work items from the primary queues in a round-robin fashion, and only if there are no work items in the set of primary queues does it pull a work item from one of the secondary queues. This allows excess work from other queues to be run on workers that don't have any other pieces of work, while still allowing prioritization of important work, as specified by queues.
I'm sure there are many more complex prioritization schemes, but separating sets of queues as "primary"/"secondary" would fulfill all fo my use cases.
