crane icon indicating copy to clipboard operation
crane copied to clipboard
verified publisher icon ipetkov

Can't build workspace dependency from git repo with relative path references in Cargo.toml

Open mpontus opened this issue 3 years ago • 2 comments

I'm trying to use crane to build a package dependent on on polkadot-runtime which is included in https://github.com/paritytech/polkadot/ repo and depends on several packages available in the same workspace.

From runtime/polkadot/Cargo.toml:

xcm = { package = "xcm", path = "../../xcm", default-features = false }
xcm-executor = { package = "xcm-executor", path = "../../xcm/xcm-executor", default-features = false }
xcm-builder = { package = "xcm-builder", path = "../../xcm/xcm-builder", default-features = false }

This presents a problem when, while building package dependencies, crane calls downloadCargoPackageFromGit to create a flat directory source containing all crates found in the git repo:

runCommandLocal "cargo-git" deps ''
  mkdir -p $out
  while read -r cargoToml; do
    local crate=$(toml2json <"$cargoToml" | \
      jq -r 'select(.package != null) | .package | "\(.name)-\(.version)"'
    )
    if [ -n "$crate" ]; then
      local dest="$out/$crate"
      cp -r "$(dirname "$cargoToml")" "$dest"
      chmod +w "$dest"
      echo '{"files":{}, "package":null}' > "$dest/.cargo-checksum.json"
    fi
  done < <(find ${repo} -name Cargo.toml)
''

Because the directory source does not reflect the structure of the original repository, create will not be able to resolve dependencies specified as relative paths, which in the following error when trying to build crate that depends on one of the workspace packages fetched from git:

error: failed to get `pallet-xcm` as a dependency of package `polkadot-runtime v0.9.24 (/nix/store/96wd59czjxqc7h7hka0mzl8a6qi7ld45-vendor-cargo-deps/fee86dd00aa0134a06a01f9b456af966945d77ffcf28dceb39cbd32320835532/polkadot-runtime-0.9.24)`
  ... which satisfies path dependency `wasm-project` of package `polkadot-runtime-wasm v1.0.0 (/build/dummy-src/target/release/wbuild/polkadot-runtime)`

Caused by:
failed to load source for dependency `pallet-xcm`

Caused by:
Unable to update /nix/store/96wd59czjxqc7h7hka0mzl8a6qi7ld45-vendor-cargo-deps/xcm/pallet-xcm

Caused by:
failed to read `/nix/store/96wd59czjxqc7h7hka0mzl8a6qi7ld45-vendor-cargo-deps/xcm/pallet-xcm/Cargo.toml`

Caused by:
No such file or directory (os error 2)

I hope this information is enough to identify the problem, but I can also draft a minimal reproducible example if it helps.

I have only brief familiarity with rust toolchain and development, but I'd happy to follow up on my diagnosis with a solution if you would point me in the right direction.

Thanks.

mpontus avatar Aug 15 '22 13:08 mpontus

Here are some of the contents of path referenced above created by vendorGitDeps and containing directory sources corresponding to registries and git repositories from the original package:

$ ls -l /nix/store/96wd59czjxqc7h7hka0mzl8a6qi7ld45-vendor-cargo-deps/
total 4
lrwxrwxrwx 1 root root   53 Jan  1  1970 138a0dc8a382e8ddb64b4a49e647445320bff2f58e3d79bd329fd32855ca5f3f -> /nix/store/1abvc7r107y0qjqyls1w2233gac1mx08-cargo-git
lrwxrwxrwx 1 root root   53 Jan  1  1970 7d84a830ef8d6e1b48acba00107ad6f3b611cd379593ab81d2d4852adc134a50 -> /nix/store/ar8yrk7m7pphmillw4i3j02ccg6wnb24-cargo-git
lrwxrwxrwx 1 root root   53 Jan  1  1970 870f227e71d876f6fda87cfb80f451f1cb3a0754fc8c1e8e8265d39382d56a3a -> /nix/store/xpf8wbdyv11i98vcbf6x1593isirpz92-cargo-git
lrwxrwxrwx 1 root root   53 Jan  1  1970 8c9ab3e925841deec677f2680029b235516752f485602cd06f168a4b6842812a -> /nix/store/qpgbqlwhwl5w6pz8pcmf8zlxmj4hn91c-cargo-git
lrwxrwxrwx 1 root root   59 Jan  1  1970 c19b7c6f923b580ac259164a89f2577984ad5ab09ee9d583b888f934adbbe8d0 -> /nix/store/jn62iwrbm3931pm1w7rn22ad7qll8pqr-vendor-registry
-r--r--r-- 1 root root 3771 Jan  1  1970 config.toml
lrwxrwxrwx 1 root root   53 Jan  1  1970 ec6ad6f8c3dd0e803fa33200329867201eec8b30b8d42e1355fcd8f0c63bcec4 -> /nix/store/ljyjqik106hqdfy2l6kqg3f2az7b3634-cargo-git
lrwxrwxrwx 1 root root   53 Jan  1  1970 ef629d17541a79f0bc18f008cfdda87cf78f321d1d131842703d21c7e52937b8 -> /nix/store/q1kw8q8pg35dhbhk1nvr9xqxy434c3kq-cargo-git
lrwxrwxrwx 1 root root   53 Jan  1  1970 fee86dd00aa0134a06a01f9b456af966945d77ffcf28dceb39cbd32320835532 -> /nix/store/zvqjk8skbx4nbjb4xknv2936m4xhaqgd-cargo-git
$ ls -l /nix/store/96wd59czjxqc7h7hka0mzl8a6qi7ld45-vendor-cargo-deps/fee86dd00aa0134a06a01f9b456af966945d77ffcf28dceb39cbd32320835532/
total 516
...
dr-xr-xr-x  9 root root 4096 Jan  1  1970 xcm-0.9.24
dr-xr-xr-x  4 root root 4096 Jan  1  1970 xcm-builder-0.9.24
dr-xr-xr-x  4 root root 4096 Jan  1  1970 xcm-executor-0.9.24
...
$ cat /nix/store/96wd59czjxqc7h7hka0mzl8a6qi7ld45-vendor-cargo-deps/config.toml
...
[source.nix-sources-fee86dd00aa0134a06a01f9b456af966945d77ffcf28dceb39cbd32320835532]
directory = "/nix/store/96wd59czjxqc7h7hka0mzl8a6qi7ld45-vendor-cargo-deps/fee86dd00aa0134a06a01f9b456af966945d77ffcf28dceb39cbd32320835532"
[source."https://github.com/paritytech/substrate?branch=polkadot-v0.9.24"]
replace-with = "nix-sources-870f227e71d876f6fda87cfb80f451f1cb3a0754fc8c1e8e8265d39382d56a3a"
git = "https://github.com/paritytech/substrate"
branch = "polkadot-v0.9.24"
...

mpontus avatar Aug 15 '22 13:08 mpontus

Thanks for the report! I have two follow up questions:

  1. Could you please provide either a link to a flake which exhibits the issue or a minimal reproduction of it?
  2. If you were to manually run cargo vendor and use its configuration, does the project build successfully? If yes we should compare the differences in how it structures crates vs how crane does it

ipetkov avatar Aug 15 '22 15:08 ipetkov

@ipetkov Here's the flake that's failling on nix build: https://github.com/mpontus/centrifuge-chain/blob/parachain/flake.nix

Running cargo vendor also creates a flat directory source, which seemingly contains all crates from git sources:

$ ls -l vendor/
drwxr-xr-x 3 mpontus users 4096 Aug 16 13:00 polkadot-runtime
...
drwxr-xr-x 3 mpontus users 4096 Aug 16 13:00 xcm
drwxr-xr-x 4 mpontus users 4096 Aug 16 13:00 xcm-builder
drwxr-xr-x 3 mpontus users 4096 Aug 16 13:00 xcm-emulator

And, after updating .cargo/config.toml, it fails with the exact same error when running cargo build:

error: failed to run custom build command for `polkadot-runtime v0.9.24 (https://github.com/paritytech/polkadot?branch=release-v0.9.24#22836e55)`

Caused by:
  process didn't exit successfully: `/home/mpontus/repos/centrifuge-chain/target/debug/build/polkadot-runtime-c62d0893e6a6adc4/build-script-build` (exit status: 1)
  --- stdout
  Information that should be included in a bug report.
  Executing build command: "/home/mpontus/.rustup/toolchains/nightly-2022-05-09-x86_64-unknown-linux-gnu/bin/cargo" "rustc" "--target=wasm32-unknown-unknown" "--manifest-path=/home/mpontus/repos/centrifuge-chain/target/debug/wbuild/polkadot-runtime/Cargo.toml" "--color=always" "--profile" "release"
  Using rustc version: rustc 1.62.0-nightly (cb1219871 2022-05-08)


  --- stderr
  error: failed to get `pallet-xcm` as a dependency of package `polkadot-runtime v0.9.24 (/home/mpontus/repos/centrifuge-chain/vendor/polkadot-runtime)`
      ... which satisfies path dependency `wasm-project` of package `polkadot-runtime-wasm v1.0.0 (/home/mpontus/repos/centrifuge-chain/target/debug/wbuild/polkadot-runtime)`

  Caused by:
    failed to load source for dependency `pallet-xcm`

  Caused by:
    Unable to update /home/mpontus/repos/centrifuge-chain/xcm/pallet-xcm

  Caused by:
    failed to read `/home/mpontus/repos/centrifuge-chain/xcm/pallet-xcm/Cargo.toml`

  Caused by:
    No such file or directory (os error 2)
warning: build failed, waiting for other jobs to finish...
error: failed to run custom build command for `kusama-runtime v0.9.24 (https://github.com/paritytech/polkadot?branch=release-v0.9.24#22836e55)`

Caused by:
  process didn't exit successfully: `/home/mpontus/repos/centrifuge-chain/target/debug/build/kusama-runtime-967cb7ba63736667/build-script-build` (exit status: 1)
  --- stdout
  Information that should be included in a bug report.
  Executing build command: "/home/mpontus/.rustup/toolchains/nightly-2022-05-09-x86_64-unknown-linux-gnu/bin/cargo" "rustc" "--target=wasm32-unknown-unknown" "--manifest-path=/home/mpontus/repos/centrifuge-chain/target/debug/wbuild/kusama-runtime/Cargo.toml" "--color=always" "--profile" "release"
  Using rustc version: rustc 1.62.0-nightly (cb1219871 2022-05-08)


  --- stderr
      Blocking waiting for file lock on package cache
  error: failed to get `kusama-runtime-constants` as a dependency of package `kusama-runtime v0.9.24 (/home/mpontus/repos/centrifuge-chain/vendor/kusama-runtime)`
      ... which satisfies path dependency `wasm-project` of package `kusama-runtime-wasm v1.0.0 (/home/mpontus/repos/centrifuge-chain/target/debug/wbuild/kusama-runtime)`

  Caused by:
    failed to load source for dependency `kusama-runtime-constants`

  Caused by:
    Unable to update /home/mpontus/repos/centrifuge-chain/vendor/kusama-runtime/constants

  Caused by:
    failed to read `/home/mpontus/repos/centrifuge-chain/vendor/kusama-runtime/constants/Cargo.toml`

  Caused by:
    No such file or directory (os error 2)

mpontus avatar Aug 16 '22 13:08 mpontus

And, after updating .cargo/config.toml, it fails with the exact same error when running cargo build:

Ah thanks for checking this, in that case this sounds like an issue with the upstream project. When using a git-dependency cargo happens to check out the entire directory and so the build script gets lucky in assuming where other crates are located on the file system, but I believe this is an incorrect assumption. If the crate was published on crates.io and consumed from there I believe the same issue would occur.

I'm inclined to close this as an upstream problem given that there isn't much we can do to change how vendored crates are consumed by cargo, but feel free to reopen if there are other ideas.

ipetkov avatar Aug 27 '22 17:08 ipetkov

@ipetkov Understood, thanks for looking into this

mpontus avatar Aug 29 '22 09:08 mpontus

@mpontus how you solved issue? we can build polkadot like this

{ pkgs, rust-nightly }:
with pkgs;
rustPlatform.buildRustPackage rec {
  # HACK: break the nix sandbox so we can build the runtimes. This
  # requires Nix to have `sandbox = relaxed` in its config.
  # We don't really care because polkadot is only used for local devnet.
  __noChroot = true;
  name = "polkadot-v${version}";
  version = "0.9.27";
  src = fetchFromGitHub {
    repo = "polkadot";
    owner = "paritytech";
    rev = "v${version}";
    hash = "sha256-LEz3OrVgdFTCnVwzU8C6GeEougaOl2qo7jS9qIdMqAM=";
  };
  cargoSha256 = "sha256-6y+WK2k1rhqMxMjEJhzJ26WDMKZjXQ+q3ca2hbbeLvA=";
  doCheck = false;
  buildInputs = [ openssl zstd ];
  nativeBuildInputs = [ rust-nightly clang pkg-config ]
    ++ lib.optional stdenv.isDarwin
    (with darwin.apple_sdk.frameworks; [ Security SystemConfiguration ]);
  LD_LIBRARY_PATH =
    lib.strings.makeLibraryPath [ stdenv.cc.cc.lib llvmPackages.libclang.lib ];
  LIBCLANG_PATH = "${llvmPackages.libclang.lib}/lib";
  PROTOC = "${protobuf}/bin/protoc";
  ROCKSDB_LIB_DIR = "${rocksdb}/lib";
  meta = { mainProgram = "polkadot"; };
}

but cumulus fails exactly with same error:

{ pkgs, rust-nightly }:
with pkgs;
let
  branch = "polkadot-v0.9.27";
  paritytech-cumulus = fetchFromGitHub {
    repo = "cumulus";
    owner = "paritytech";
    rev = branch;
    hash = "sha256-nbHdXv/93F6vHXWr/r9+AqvBBa5f9L6tmoIs8EEqiKM=";
  };
  substrate-build-attrs = {
    buildInputs = [ openssl zstd ];
    nativeBuildInputs = [ clang openssl pkg-config ]
      ++ lib.optional stdenv.isDarwin
      (with darwin.apple_sdk.frameworks; [ Security SystemConfiguration ]);
    LD_LIBRARY_PATH = lib.strings.makeLibraryPath [
      stdenv.cc.cc.lib
      llvmPackages.libclang.lib
    ];
    LIBCLANG_PATH = "${llvmPackages.libclang.lib}/lib";
    PROTOC = "${protobuf}/bin/protoc";
    ROCKSDB_LIB_DIR = "${rocksdb}/lib";
  };
  dependencies = rust-nightly.buildDepsOnly
    (substrate-build-attrs // { src = paritytech-cumulus; });
in with pkgs;
rust-nightly.buildPackage (substrate-build-attrs // rec {
  __noChroot = true;
  name = "statemine-v${version}";
  cargoArtifacts = dependencies;
  version = branch;
  src = paritytech-cumulus;
  cargoSha256 = "sha256-s5i+XqQH+FHCqG/L/gI9BWNKDMxxLwZ/tbeFD68hNew=";
  doCheck = false;
  meta = { mainProgram = "polkadot-parachain"; };
  cargoBuildCommand = "cargo build --release --locked --bin polkadot-parachain";
  installPhase = ''
    mkdir -p $out/bin
    cp target/release/polkadot-parachain $out/bin/polkadot-parachain
  '';
})

dzmitry-lahoda avatar Oct 04 '22 18:10 dzmitry-lahoda

@mpontus try use pkgs.rustPlatform.buildRustPackage, not crane. it worked in ComposableFi/composable. run with sandbox relaxed and __noChroot.

so for some reason failed to build statemine from cumulus like that. for cumulus just used usual derivation with calls to rust compiler. it taks time so as deps are no chached.

dzmitry-lahoda avatar Oct 04 '22 20:10 dzmitry-lahoda

@ipetkov would be rigth to say if pkgs.rustPlatform.buildRustPackage can build X then crane can build X because crane is superset of rustPlatform for building, with assumption that patches and substitute or full swap of pipeline not used.

dzmitry-lahoda avatar Oct 05 '22 00:10 dzmitry-lahoda

would be rigth to say if pkgs.rustPlatform.buildRustPackage can build X then crane can build X because crane is superset of rustPlatform for building, with assumption that patches and substitute or full swap of pipeline not used.

Crane is an implementation that is completely distinct from rustPlatform.buildRustPackage and I cannot guarantee that they will be bug-for-bug compatible with each other.

I would imagine that the majority of cargo projects can successfully be built with both, but if a project's build script is not sandbox friendly (e.g. requires disabling chroot, or requires assuming that the crates are checked out from cargo's index in $CARGO_HOME) there will be friction trying to get things building

ipetkov avatar Oct 08 '22 20:10 ipetkov