Crate owner based build backoff

[Kixiron]

Really big releases can swamp the build queue and make normal people wait hours for their crate to get built and manual de-prioritization requires human intervention, so some sort of owner-based backoff would be a good automatic solution. The basic idea is that if over n crates with the same owner are present within the build queue at the same time, x0 of their crates get the -1 priority level, x1 crates get the -2 priority level, ad nausea. Basically, it gradually pushes back more and more crates based on how many an owner is currently building so that it's fair to everyone, the owner still gets some of their crates built within a timely manner and all of the normal people get pushed to the front of the line. It also prevents the (terrifying) scenario where two or more biggies are published at the same time, so instead of the first to publish getting all of their crates built and the latter ones waiting hours for any of their crates to build, they both actually get some crates built within a reasonable time span. Additionally, it also doesn't punish big publishers who just want to make a patch release of some sort, if people-who-publish-50-crates-at-a-time want to do a one-off random release for a patch or something they aren't permanently in a de-prioritized state even though they've only released one single crate (opposed to the manual de-prioritization where they are unconditionally pushed back in the queue)

[ckaran]

Quoting my own post:

OK, I looked at the documentation for docs.rs, and it looks like each crate is built within its own docker image. So this actually gives us a really simple procedure for dealing with the issue that allows us to reweight the relative performance of each build on the fly:

1. On start up, use docker update -c to set the CPU share to 8. (see this article for an explanation of the CPU share option).
2. For every 15 minutes of runtime of a given container, decrease the container's CPU share by 1, to a minimum of 1.
3. There is no step 3.

Basically, this is kind of like using renice, but on the container as a whole, which means that resources are idle only if there's no work to do.

[syphar]

I don't know the exact CPU-share config, but:

we already have a hard limit of 15 minutes build-time (with some exceptions)

[ckaran]

Yup, this is a way of dealing with actors that use more than their fair share, without stopping them entirely. Basically, if a crate is built very fast, its CPU share will be high, and it will get through the queue quickly. However, if a build takes a long time, it eventually ends up being given lower priority (not zero priority), so it will eventually finish, but it won't slow down small & fast crates.

[jyn514]

@ckaran your idea only works if multiple builds are run in parallel, because cpu-share is just a percentage of CPU and has no meaning in absolute terms. But they aren't; builds are run one at a time until completion, so the CPU-share has no effect at all.

[jyn514]

Also, to anyone visiting this post from internals, please reread @pietroalbini's write-up of why we are not currently dedicating time to making the build queue more fair: https://internals.rust-lang.org/t/docs-rs-build-prioritization/15511/13?u=jyn514

[ckaran]

@ckaran your idea only works if multiple builds are run in parallel, because cpu-share is just a percentage of CPU and has no meaning in absolute terms. But they aren't; builds are run one at a time until completion, so the CPU-share has no effect at all.

My apologies! I thought the build system was run concurrently, if not, then you're right, this won't work at all...

[kdy1]

I second this. Because of the way cargo works, a breaking change of swc_common means breaking changes of all swc crates. But that's not an api change of crates. It's to guarentee a successful version resolving for rust-side users.

So if swc uses many slots in the build queue, it means a breaking change of some important upstream crate and showing docs for the old version is perfectly fine.