Build targets with several versions of the same library

[Leandros]

I'm building a project which has the same library as a dependency several times, due to being required by my dependencies in several versions.

This results in an error from cargo-chef:

 > [builder 3/6] RUN cargo chef cook --release --target x86_64-unknown-linux-musl --recipe-path recipe.json:
#14 1.004 error: failed to parse lock file at: /app/Cargo.lock
#14 1.004
#14 1.004 Caused by:
#14 1.004   package `getrandom` is specified twice in the lockfile
#14 1.005 thread 'main' panicked at 'Exited with status code: 101', /usr/local/cargo/registry/src/github.com-1ecc6299db9ec823/cargo-chef-0.1.31/src/recipe.rs:145:27
#14 1.005 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

getrandom is indeed multiple times in my lockfile due to it being required by several versions of rand

[bsod90]

Hi, @Leandros ! I just ran into the same problem and I'm wondering if you found a good workaround yet. Thanks! :)

[LukeMathWalker]

Hey! Can you provide a small project where I can reproduce the issue?

[bsod90]

Hey, @LukeMathWalker ! Thanks for looking into this! I believe it's quite simple to reproduce: all you need is two top-level dependencies that internally depend on different versions of the same package. For example, I just created a Cargo.toml

[package]
name = "test_rust"
version = "0.1.0"
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
num-integer = "0.1.44"
rand = "0.6.5"

Which resulted in Cargo.lock mentioning two different versions of autocfg. I hope this helps :)

[Leandros]

Hi, @Leandros ! I just ran into the same problem and I'm wondering if you found a good workaround yet. Thanks! :)

@bsod90 Sadly I could not figure out a solution.

[asportnoy]

Has anyone found a workaround for this?

[rcastill]

Same here. In my case the conflicting dependencies are:

ring v0.16.20  <- spin v0.5.2
flume v0.10.9  <- spin v0.9.2

I can confirm that cargo buildworks outside and inside dockerfile if I don't use cargo-chef.

[rcastill]

@LukeMathWalker I found the problem in my case. Hopefully it will be useful for the rest.

Hunt

In order to find the error, I changed the source code of cargo-chef to debug. Finally I arrived to the conclusion that skeleton::version_masking::parse_local_crate_names was returning ALL of the dependencies names in my project (including external ones -- in fact, I don't have local crates in this project).

Cause

I'm using some private crates (using git ssh); in order to make that work in the container, I first CARGO_HOME=$(pwd)/.cargo cargo fetch in the project root, and then copy all the downloaded crates into the container (no need to pass ssh keys into the container). Since parse_local_crate_names searches [package] section in ALL Cargo.toml found in the project recurively, that included the external dependencies inside .cargo as well.

Solution

• DO NOT copy .cargo folder before cargo chef prepare
• DO set CARGO_HOME=$(pwd)/.cargo and copy .cargo folder before cargo chef cook

[pymongo]

same problem occur on my jenkins CI, my solution is clean up jenkins docker cache and wip out workspace(.docker/cache), then build again would success

[albx79]

I'm having the same problem but I can't understand the workarounds.

In my case the project is a rather large workspace consisting of a dozen crates, and many many transitive dependencies. It's unavoidable that some external crate will appear twice with different versions, and it's not a problem for a normal cargo build. However, running cargo chef cook in docker will fail with the posted error.

@rcastill could you provide a minimal Dockerfile where you use the trick of copying the .cargo folder? Thanks.

[pwlodarczyk92]

@albx79 Multiple versions of the same dependency is not a problem in itself. The problem is that cargo chef may incorrectly interpret such dependency as a dependency on a local crate. Cargo chef modifies all local crates' versions to 0.0.1 so that changing the version of your crate doesn't invalidate docker layer cache that contains compiled, external dependencies.

This way, the mock Cargo.lock generated by cargo chef will contain multiple instances of the same, external dependency with the same 0.0.1 version. The package getrandom is specified twice in the lockfile is, AFAIK, an error emitted by cargo build that's not specific enough and adds to the confusion.

Cargo chef may incorrectly interpret a dependency as local one when it's Cargo.toml file appears in the same directory where you prepare your recipe, even if it's not part of your workspace.

In my case, the problem occured because I was creating a recipe in the root directory of the docker image. This caused cargo-chef to interpret all Cargo.toml files in the whole filesystem (e.g. the ones cached by cargo during installation of cargo chef itself) as local dependencies. Moving the cargo chef prepare to a separate directory fixed the problem, as it does not contain other Cargo.toml files than the ones in your project:

FROM rust:1.64-bullseye AS planner
WORKDIR plan
RUN cargo install --locked cargo-chef
COPY . .
RUN cargo chef prepare --bin ...

without WORKDIR plan I got the same error as everyone else when I tried to cook the recipe.

There should be some explicit warning on this behavior of cargo-chef in the readme and the cargo build error should be fixed too, I guess.

[LukeMathWalker]

This should no longer be the case since we now look exclusively at the crates in the current workspace as local (via cargo metadata) instead of traversing the filesystem manually looking for Cargo.toml files.

I'm going to close this as resolved, feel free to re-open if you run into the same issue again.