Support workspace dependencies

[ipetkov]

Rust 1.64 supports declaring cargo dependencies at the workspace level. We should take those into account when vendoring crates

https://blog.rust-lang.org/2022/09/22/Rust-1.64.0.html

[tomhoule]

Just to start a discussion on this, since we are running into this at Prisma and we want to start using crane. I tried using crane with a repo with workspace dependencies.

In the repo itself, no problem. It looks like cargo transparently does the job.

In another repo that depends on the repo with workspace dependencies via git, here's what we get:

@nix { "action": "setPhase", "phase": "unpackPhase" }
unpacking sources
unpacking source archive /nix/store/2yiwx5262v6hj2z17s0rg2fdd7jrdhqr-dummy-src
source root is dummy-src
@nix { "action": "setPhase", "phase": "patchPhase" }
patching sources
Executing configureCargoCommonVars
@nix { "action": "setPhase", "phase": "configurePhase" }
configuring
will append /build/dummy-src/.cargo-home/config.toml with contents of /nix/store/b3p3fckc4jx9x20xd8i2y9qydmky7bp7-vendor-cargo-deps/config.toml
default configurePhase, nothing to do
@nix { "action": "setPhase", "phase": "buildPhase" }
building
++ command cargo --version
cargo 1.64.0 (387270bc7 2022-09-16)
++ command cargo check --target=wasm32-unknown-unknown
error: failed to get `prisma-fmt` as a dependency of package `prisma-fmt-build v0.1.0 (/build/dummy-src)`

Caused by:
  failed to load source for dependency `prisma-fmt`

Caused by:
  Unable to update https://github.com/prisma/prisma-engines#18aec693

Caused by:
  failed to update replaced source https://github.com/prisma/prisma-engines#18aec693

Caused by:
  failed to parse manifest at `/nix/store/b3p3fckc4jx9x20xd8i2y9qydmky7bp7-vendor-cargo-deps/a0c466a33f02e3dbd1af7484529b17f40ec298f145519133ca66603a48fab109/native-types-0.1.0/Cargo.toml`

Caused by:
  failed to find a workspace root

I interpret this as cargo looking for the workspace Cargo.toml to resolve the dependencies of vendored crates, but since the workspace crates are vendored individually, the workspace root Cargo.toml is not available.

I'm not super familiar with crane internals, so it may not be a good idea, but could the fix be as simple as detecting which crates use workspace dependencies in vendorGitDeps.nix, then copying the workspace dependencies from the root Cargo.toml in there so the vendored deps have all they need?

[ipetkov]

@tomhoule could you provide a flake.nix with which I can reproduce the issue? (bonus points if it's a minimal setup)

[ipetkov]

Wonder if this is an issue with cargo itself. Looks like cargo vendor does not (yet?) work with dependencies which use workspace inheritance: https://github.com/rust-lang/cargo/issues/11192

We don't use cargo vendor directly, but its behavior is a reference implementation for sure. If there is a way to massage the crate sources to make cargo work I'm all for it, but I don't know off the top of my head what that format would be

[tomhoule]

Here's the minimal setup: https://github.com/tomhoule/crane-workspace-deps-repro — it depends on a large workspace so it takes a while to vendor the dependencies but it's minimal otherwise. I expect the result would be the same replacing the dependency on prisma-fmt with any git dependency with workspace dependencies.

[dzmitry-lahoda]

same issue

composable-deps> setting CARGO_BUILD_RUSTFLAGS to "--remap-path-prefix /nix/store/yp5hrdck42621s2n4nngzwj9a9n4mg22-vendor-cargo-deps=/sources"
composable-deps> building
composable-deps> ++ command cargo --version
composable-deps> cargo 1.67.0-nightly (16b097879 2022-11-14)
composable-deps> ++ command cargo build --profile release
composable-deps> error: failed to get `ibc` as a dependency of package `pallet-cosmwasm v0.0.1 (/tmp/nix-build-composable-deps-2.3.4.drv-0/dummy-src/parachain/frame/cosmwasm)`
composable-deps> Caused by:
composable-deps>   failed to load source for dependency `ibc`
composable-deps> Caused by:
composable-deps>   Unable to update https://github.com/ComposableFi/centauri/?rev=9d233782725943db8e992107cbd1edb0a4e0133e#9d233782
composable-deps> Caused by:
composable-deps>   failed to update replaced source https://github.com/ComposableFi/centauri/?rev=9d233782725943db8e992107cbd1edb0a4e0133e#9d233782
composable-deps> Caused by:
composable-deps>   failed to parse manifest at `/nix/store/yp5hrdck42621s2n4nngzwj9a9n4mg22-vendor-cargo-deps/cdbb7e1ee239d36bd2cd565fe332c09be3506881d778b676255fcbd53cf1b1f6/ics11-beefy-0.1.0/Cargo.toml`
composable-deps> Caused by:
composable-deps>   failed to find a workspace root
error: builder for '/nix/store/6zbnk9n39gsffh96p6dh2xmi95lmnn56-composable-deps-2.3.4.drv' failed with exit code 101
Stopped watching /nix/store and waiting for queue to empty ...
Done.
Error: Process completed with exit code 1.

So we ComposableFi/composable depends on ComposableFi/centauri which started to use workspace deps.

If there will be donated button in crypto can donate the equivalent of 42 USD for fix.

[dzmitry-lahoda]

We use latest crane git commit.

[tomhoule]

There's progress on the cargo vendor issue: https://github.com/rust-lang/cargo/pull/11414