cargo

cargo copied to clipboard

I was wondering if anyone has contemplated somehow sharing compiled crates. If I have a number of projects on disk that often have similar dependencies, I'm spending a lot of time recompiling the same packages. (Even correcting for features, compiler flags and compilation profiles.) Would it make sense to store symlinks in ~/.cargo or equivalent pointing to compiled artefacts?

There's been musings about this historically but never any degree of serious consideration. I've always wanted to explore it though! (I think it's definitely plausible)

sccache is one option here - it has a local disk cache in addition to the more exotic options to store compiled artifacts in the cloud.

sccache would be good for the compilation time part, but it'd be nice to also get a handle on the disk size part of it.

cc https://github.com/rust-lang/cargo/issues/6229

I think you can put

[build]
target-dir = "/my/shared/target/dir"

in ~/.cargo/config.

But I have no idea if this mode is officially supported. Is it?

Yes it is, as is setting it with the corresponding environment variable. However the problems with cargo never deleting the unused artifacts gets to be dramatic quickly. Hence the connection to #6229

@joshtriplett and I had a brainstorming session on this at RustNL last week.

It'd be great if cargo could have a very small subset of sccache's logic: per-user caching of intermediate build artifacts. By building this into cargo, we can tie it into all that cargo knows and cane make extensions to better support it.

Risks

• Poisoning the cache from
  • Broken builds (e.g. from incremental compilation bugs)
  • Non-deterministic builds
  • mtime bugs
• Races with parallel builds
  • Performance hits from locking
• Running out of disk space

To mitigate problems with cache poisoning

• Packages must advertise that they are deterministic
  • Long term: Wasm build scripts and proc-macros to ensure determinism
• Initially limit the caching to packages from immutable sources
  • No mtime bugs
  • No need for incremental compilation

As a contingency for if the cache is poisoned, we need a way to clear the cache (see also #3289)

To mitigate running out of disk space, we need a GC / prune (see also #6509)

• One strategy is to clear caches for a rust version that is no longer installed
• We could manually track atime

Locking strategy to mitigate race conditions / locking performance

• Assumptions:
  • Parallel builds are likely
  • Parallel builds of the same package fingerprint are unlikely
• Design:
  1. Does it exist (without lock)
  2. Build outside the cache
  3. If can't do atomic rename, move into a temp dir, then do an atomic rename. If it now exists, just delete the tmpdir
• Read/prune lock
  • Multi-Reader/single-writer lock
  • What about reading it?
    • Hold lock wile building, blocking prunes
    • Could copy out while holding lock to minimize lock time
    • Hold a lock for copy it
    • Or don't bother copying, just block the prune until lock is available
  • Prune by renaming and then deleting

Transition plan (modeled off of sparse registry)

• Steps:
  1. Unstable
  2. Hacky env variable to opt-in (unstable)
  • On stable, warn if env variable is set so people can set it globally and use versions of Rust from Step 2 and Step 3
  4. Hacky env variable to opt-in (stable)
  5. Stable
• Don't need full cleanup/pruning strategy until it is stable

Wonder if something like reflink would be useful

See also #7150

Some complications that came up when discussing this this with ehuss.

First, some background. We track rebuilds in two ways. The first is we have an external fingerprint that is a hash that we use to tell when to rebuild. The second is we have the hash of build inputs we pass to rustc with -Cmetadata that is used to keep symbols unique. We include this in the file name, so if -Cmetadata changes, then the filename changes. If the file doesn't exist, that is a sure sign it needs to be built.

Problems

• We co-mingle files in the target directory.
  • This makes it easy for us to pass a single directory for rustc to slurp up rlibs
  • Some other tools depend on this for slurping up rlibs or for asm output from rustc
  • This is a problem because we'll need all artifacts for an immutable package to be in isolated directories, for capturing the files and reading from the cache
• RUSTFLAGS is only present in the fingerprint and not in -Cmetadata (#8716).
  • We don't want PGO related RUSTFLAGS to change symbols
  • We don't want remap related RUSTFLAGS to change symbols
  • However, this introduces mutable data into the immutable package, making it so we can't cache it

I guess the first question is whether the per-user cache should be organized around fingerprint, -Cmetadata, or something else. Well, -Cmetadata isn't an option so long as it doesn't have RUSTFLAGS, which it shouldn't, so it would more be fingerprint or us adding a new hash type, one that maybe we reuse with the file names and ensure doesn't cause problems with rustc.

Cargo uses relative paths to workspace root for path dependencies to generate stable hashes. This causes an issue (https://github.com/rust-lang/cargo/issues/12516) when sharing target directories between package with the same name and version and relative path to workspace.

For me, the biggest thing that needs to be figured out before any other progress is worth it is how to get a reasonable amount of value out of this cache.

Take my system

• 74 repos with a Cargo.lock in the root
• 65 of those repos have syn in the lockfile
• Among those 65 repos, 44 different versions of syn are used (there is a mix of v1 and v2)

This is a "bottom of the stack" package. As you go up the stack, the impact of version combinations grows dramatically.

I worry a per-user cache's value will only be slightly more than making cargo clean && cargo build faster and that doesn't feel worth the complexity to me.

How did you do that analysis? I'd be interested in running it on my own system.

Also. re caching and RUSTFLAGS, could it be an optional to simply fall back to the existing caching scheme (project-specific target dir) if RUSTFLAGS is set at all? Personally, I work on very few projects that utilize RUSTFLAGS (AFAIK.. although I also have it configured globally right now, with -C link-arg=-fuse-ld=mold, which I'd have to disable or find an alternative solution for), so everything else benefitting from a shared cache dir might already be useful.

How did you do that analysis? I'd be interested in running it on my own system.

Pre-req: I keep all repos in a single folder.

$ ls */Cargo.lock | wc -l
$ rg 'name = "syn"' */Cargo.lock -l | wc -l
$ rg 'name = "syn"' */Cargo.lock -A 1 | rg version | rg -o '".*"' | sort -u | wc -l

(and yes, there are likely rg features I can use to code-golf this)

Thanks! I keep my projects in two dirs (approximated active and inactive projects), but running this separately on both I get the following. Also included futures-util as another commonly-used crate, one that does not get released as often.

stat	active	inactive
number of crates / workspaces	34	91
workspaces pulling in syn	29	72
different versions of syn	15	33
workspaces pulling in futures-util	20	44
different versions of futures-util	3	14

Also syn is part of a set of crates that gets a lot of little bumps. This is common for dtolnay crates, but not so much for a whole host of other crates -- so I'm not sure this particular test is very representative. (Note that I'm definitely not disagreeing that the utility of a per-user compiled artifact cache might not be as great as hoped.)

FWIW, given what I see Rust-Analyzer doing in a large workspace at work (some 670 crates are involved) it seems to be doing a lot of recompilation even with only weekly updates to the dependencies so even within a single workspace there might be some wins?

Somebody with a deduplicating file system could share their statistics for a theoretical upper bound?

@djc yes, syn bumps a lot (I wish more did that personally). I chose it partly because of the recent proc-macro conversions and the fact that by being a bottom-of-the-stack crate, it would cause cache missing for everything above it. This highlights the problem of dependencies causing cache misses.

I might be missing something obvious, but what should be the behavior of cargo clean? Should it remove all artifacts, only those that would be otherwise used by the current crate, or do nothing at all?

I'm asking because if cargo clean in one crate would increase compile time in an unrelated crate by removing a compiled artifact, then this looks like a negative side-effect (compared to the positive side-effect of compiling one crate reducing the compile time of an unrelated crate because they happen to share an artifact).

My expectation is that cargo clean would clean your target/ directory and not touch the cache. We are looking at adding a GC for global resources which would then also apply to the cache. Depending on feedback, maybe we'd explore other aspects.

Thanks! This looks reasonable to me. The only possible issue I see is when cache corruption occurs (either because of tooling like rust-analyzer or ephemeral hardware issue). In that case, because we lost cache isolation by design, we would need to wipe everything. But as far as I'm concern, that's acceptable.

IMHO it would be nice if cargo could offload completely the caching management and just offer an interface to lookup cacheable items and provide them on misses and leave all the rest to other tools (e.g. sccache), this way cargo doesn't need to have more logic inside and people using org-wide caches can just write/use a smaller integration.

@lu-zero what do you see as "the rest" for being left to other tools?

I see a lot of the complexity involved here being determining what should be cached. While some level of complexity will be involved with GC, we'll be needing that anyways for things like "cargo script".

A downside to offloading completely is that it makes it so other users don't get caching out-of-the-box which has a big impact on how many would use caching.

Its also easier for us to iterate on the design when its all internal vs also working on a unstable interface to then stabilize. I see this as a potential stepping stone for additional caching strategies.

Right now ccache and such tools tend to interpose between the actual compiler and their caching logic is tied to to capturing the context the best they could.

Then based on that information they can lookup cached items and freshen/retrieve them with arbitrary complex storage logic/policies.

Cargo has the full picture already so finding a good way to deliver it to, e.g., sccache would allow it to cache more and potentially better and we do not have to reinvent the wheel regarding to how to manage the cache itself, distribute it across and so on and so forth.

I think that focusing on building that interface might improve the current situation for sccache users and if somebody is willing to add a tiny-cache by having the simplest cache we can later integrate in cargo instead of suggesting to install sccache.

We talked about this more in office hours.

While local development won't initially benefit from this, this could help CI a lot because the cache size would be better managed without external tools needing to shrink things.

Then when we get to plugin support, this would benefit CI even further because you could then have fine grained caching across CI jobs.

Projects could then offer read-only access to this cache so local development could be sped up.

A potential path for this

Initial: For packages that are (1) immutable (non-local) and (2) deterministic (no build.rs, no proc-macros), (3) no rustflags are used, build these in a shared target directory and have dependents lock and point to both shared directories

From there, we could do (note: this is a tree of tasks)

• Move to a directory per cached package for easier management
  • GC of these directories
  • Copy in/out of these directories to reduce the time the lock is held
• Find higher level of abstractions for rustflags or expose more narrowed scope versions (BIN_RUSTFLAGS, top level rustflags, etc)
• Allow build.rs and proc macros pinky promise that they are deterministic (don't read env variables, talk to process like sql, only read their only file system)
• Sandboxing of build.rs and proc macros to know whether they are deterministic based on the capabilities they ask for
• Refactor things to clean abstraction for the cache to make it easier to reason about
  • Create a plugin system that allows multiple active plugins much like credential process (biggest issue is knowing which one to write to)

While local development won't initially benefit from this, this could help CI a lot because the cache size would be better managed without external tools needing to shrink things.

How would CI benefit? Is the idea that only those packages that are deterministic would be stored in CI cache, so it is smaller than current solutions, while guaranteeing that it's reusable as long as dependencies don't change?

Also local development won't benefit? At all? I think there should at least be a little bit of deduplication between projects, even if it's just low-level-ish crates?

Anyways, the plan above seems like a great way to make progress. Start small and build things up gradually. Also further incentivizes making libs compile-time deterministic.

How would CI benefit? Is the idea that only those packages that are deterministic would be stored in CI cache, so it is smaller than current solutions, while guaranteeing that it's reusable as long as dependencies don't change?

I might have jumped ahead on that because keeping the cache size smaller for non-local builds (which would make CI cache upload/download faster) would require GC which would come later.

Also local development won't benefit? At all? I think there should at least be a little bit of deduplication between projects, even if it's just low-level-ish crates?

To be more precise "the amount of benefit for local development will be small enough that it would not justify this feature on its own". For any no-deps packages which have few releases, you'd get sharing but then you won't be updating them often anyways. I also worry about having that stable or being stuck that way too long because it discourages two bad practices (1) people over-optimizing for no-deps and (2) people doing fewer, larger releases

@epage where you'd put this caching layer?

Right now the caching with sccache happens at compiler invocation, with the problems and restrictions that should be well known now.

If the caching layer has its say at dependency resolution you could give priority to whatever is cached as long it fits the version constraints and potentially have more hits, if it happens post-resolution then it would be still an improvement over the status-quo.

I expect the cache to mostly work by

• When expecting to compile a package
  • If in cache, copy it into CARGO_TARGET_DIR
  • If not in cache, compile it and write it to cache

That said, there is an effort to generalize the yanked/offline/error status for potential versions to unblock experiments in allowing alternative sources affect version selection. That could then be used also as part of caching though it'd likely have limits because it wouldn't easily be able to tell if the cached item has the right fingerprint.