Cap the number of concurrent builds from the same site

[davemcorwin]

trafficstars

User Story

As a Federalist user and operator I don't want the builds for a site to be backed up behind a noisy neighbor.

Background (Optional)

Occasionally one site will receive a bunch of commits, or multiple branches (ex. from Netlify CMS) that can impact the ability of other builds to run since we only support a certain amount of concurrency.

We currently do NOT create a new build for a branch if an existing one is queued, however, we should extend this logic to ensure that only a certain number of builds for a repository can be in process at once.

Acceptance Criteria

• [ ] Concurrent builds per repository are limited to $limit
• [ ] Requested builds above $limit are moved to pending state
• [ ] Builds in pending state are queued when builds are below $limit
• [ ] Change made live via Federalist web.

Level of effort - medium

Implementation outline:

• [ ] Centralize the logic for determining how to handle a build request Currently the logic that determines how to handle a requested build if one is already queued is duplicated in 2 places:

  • https://github.com/18F/federalist/blob/staging/api/controllers/build.js#L66 (user initiated from the UI)
  • https://github.com/18F/federalist/blob/staging/api/services/Webhooks.js#L61 (system initiated from GH webhook)

  We also create builds

  • https://github.com/18F/federalist/blob/staging/api/services/Domain.js#L137 (system initiated after domain is provisioned)
  • https://github.com/18F/federalist/blob/staging/api/services/SiteCreator.js#L161 (system initiated after site added)
  • https://github.com/18F/federalist/blob/staging/api/services/ScheduledBuildHelper.js#L13 (system initiated nightly)
  • https://github.com/18F/federalist/blob/staging/api/controllers/site.js#L154 (system initiated after site update)

  So we should consider if this logic should be used in these locations as well.

• [ ] Create a new build state pending (open to other names)

• [ ] Update build request logic to create a pending build when there is not a queued build for the branch AND there are >= $limit builds in created, queued, tasked, or processing

• [ ] Create repeated job (every 10 minutes? 5 minutes?) to queue pending builds if number in progress for site is < $limit

• [ ] add environment variable for $limit and determine initial reasonable value

• [ ] add GH Issue in 18f/federalist.18f.gov to update documentation for this and possible for all build states