An example of overriding a crate with a crate in a private Git repository

[rbozan]

It's not clear to me how you'd use a private Git repository and crane. In my Cargo.toml I specified my private Git repo but when I'm trying to build I'm seeing

++ command cargo check --profile release --all-targets
    Updating git repository `ssh://[email protected]/l4010/scenario_builder.git`
warning: spurious network error (2 tries remaining): failed to resolve address for gitlab.com: Temporary failure in name resolution; class=Net (12)
warning: spurious network error (1 tries remaining): failed to resolve address for gitlab.com: Temporary failure in name resolution; class=Net (12)
error: failed to get `scenario_builder` as a dependency of package `backend v0.1.0 (/build/dummy-src)`

I guess I have to do something with lib.downloadCargoPackageFromGit but would I then need to append it to a vendor file? It's not clear to me and an example of this use case would be nice.

Some example:

{
  description = "Build a cargo project without extra checks";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";

    crane = {
      url = "github:ipetkov/crane";
      inputs.nixpkgs.follows = "nixpkgs";
    };

    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, crane, flake-utils, ... }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
        };

        craneLib = crane.lib.${system};

        # Download this from Git:
        scenario_builder-crate = craneLib.downloadCargoPackageFromGit {
          git = "test";
          rev = "main";
        };

        # And use all other crates from Cargo.lock:
        cargoDeps = craneLib.vendorCargoDeps {
          src = ./.;
        };

        # And build it but somehow override the specific crate that I downloaded from Git:
        my-crate = craneLib.buildPackage {
          src = craneLib.cleanCargoSource ./.;
          cargoVendorDir = cargoDeps;
        };

      in
      {
        checks = {
          inherit my-crate;
        };

        packages.default = my-crate;

        apps.default = flake-utils.lib.mkApp {
          drv = my-crate;
        };

        devShells.default = pkgs.mkShell {
          inputsFrom = builtins.attrValues self.checks;

          nativeBuildInputs = with pkgs; []
            protobuf
          ];

          PROTOC = "${pkgs.protobuf.out}/bin/protoc";

          shellHook = ''
            rm ./.cargo/config.toml
            ln -s ./config.local.toml ./.cargo/config.toml
          '';
        };
      });
}

[ipetkov]

Hi @rbozan thanks for the report!

Just a quick question, does the Cargo.toml file depend on a private git repository? Or are you trying to replace a dependency with a fork which happens to be behind a private git repository?

If it's the former: Nix should have a way to fetch from private git repos by using whatever ssh keys you may have available during evaluation (though that isn't the error shown above fwiw).

If it's the latter: vendorCargoDeps should automatically handle vendoring all dependencies if the git repo is listed in the Cargo.lock file (modulo Nix being able to download the files which brings us back to the first case above).

A few other comments worth mentioning based on the example flake you've provided (not sure how relevant they may be):

• scenario_builder-crate isn't referenced by anything (nor does it have a valid URL) so it for sure won't have any effect on anything
• Updating git repository ... this indicates that the dependency wasn't vendored and cargo is trying to reach it on the network (but failing due to the sandbox). Is the Cargo.lock file being patched somehow?

[rbozan]

Hey @ipetkov thanks for the help.

In the Cargo.toml I have the following:

[dependencies]
scenario_builder = { git = "ssh://[email protected]/l4010/scenario_builder.git", features = [ "graphql" ] }

Then I have a .cargo/config.toml which has the following:

[net]
git-fetch-with-cli = true

[patch."ssh://[email protected]/l4010/scenario_builder.git"]
scenario_builder = { path = "../scenario_builder/builder" }

This would patch the package to the locally checked out repository.

The Cargo.lock contains the following:

[[package]]
name = "scenario_builder"
version = "0.1.0"
dependencies = [
 "anyhow",
 "async-graphql",
 "bollard",
 "derivative",
 "futures",
 "serde",
 "serde_yaml",
]

So what solution do I need to apply here?

[ipetkov]

Thanks @rbozan , could you check one more thing for me?

If you move the patch section out of .cargo/config.toml and move it to Cargo.toml does that make any difference or do you get the same error?

[ipetkov]

If that doesn't make a difference could you also try running cargo update after moving the patch section into Cargo.toml and see if that affects anything

[rbozan]

@ipetkov

If I remove the patch entirely in my .cargo/config.toml (so I don't have a patch anywhere) it indeed managed to work after doing cargo generate-lockfile. The command added the source here:

[[package]]
name = "scenario_builder"
version = "0.1.0"
source = "git+ssh://[email protected]/l4010/scenario_builder.git#4581ee77da6f8f29cc0ed9ecb238ec5baaa5649e"
dependencies = [
 "anyhow",
 "async-graphql",
 "bollard",
 "derivative",
 "futures",
 "serde",
 "serde_yaml",
]

And then it complained about OpenSSL but at least not about the crate.

However I have the following patch in .cargo/config.toml:

[patch."ssh://[email protected]/l4010/scenario_builder.git"]
scenario_builder = { path = "../scenario_builder/builder" }

It removes the source in Cargo.lock and I'm getting this error:

> Caused by:
       >   failed to clone into: /build/dummy-src/.cargo-home/git/db/scenario_builder-4f0ba41af41e1991
       >
       > Caused by:
       >   network failure seems to have happened
       >   if a proxy or similar is necessary `net.git-fetch-with-cli` may help here
       >   https://doc.rust-lang.org/cargo/reference/config.html#netgit-fetch-with-cli
       >
       > Caused by:
       >   failed to resolve address for gitlab.com: Temporary failure in name resolution; class=Net (12)

But when I move the patch to Cargo.toml I'm getting this error:

       > error: failed to load source for dependency `scenario_builder`
       >
       > Caused by:
       >   Unable to update /build/scenario_builder/builder
       >
       > Caused by:
       >   failed to read `/build/scenario_builder/builder/Cargo.toml`
       >
       > Caused by:
       >   No such file or directory (os error 2)
       For full logs, run 'nix log /nix/store/3q94vj4v2kciqnplha4if48nyhb8qq5x-backend-deps-0.1.0.drv'.

So in the last case it seems that it couldn't find the dependency in the /build folder, I guess because it's not copied / symlinked there?

The same error message appears when I just do this, without patches:

scenario_builder = { path = "../scenario_builder/builder", features = [ "graphql" ] }

So I guess the local paths not working is a different case.

[ipetkov]

So I guess the local paths not working is a different case.

The local path is still in the root of the repo right? (or rather contained in whatever subpath is being passed in via src = ...)

The only other thing I can think of the fact that we update .cargo/config.toml so that cargo can pick up the vendored dependencies, so maybe something is going wrong there with the existing local changes.

Do you have some kind of public reproduction I can use to debug this further? The code itself isn't important, it can be stubbed out as long as the structure and cargo configs can reproduce the same issue

[rbozan]

So I guess the local paths not working is a different case.

The local path is still in the root of the repo right? (or rather contained in whatever subpath is being passed in via src = ...)

Well no. My structure is a bit like this:

Projects/
├─ my_project/
│  ├─ Cargo.toml
│  ├─ flake.nix
├─ scenario_builder/
│  ├─ Cargo.toml

[ipetkov]

@rbozan ah interesting, I think what is happening is that since the flake.nix file is inside of my_project, using src = craneLib.cleanCargoSource ./.; means only my_project is "visible" during the build.

Could you try moving scenario_builder inside of my_project and see if that works? (Alternatively you could consider moving flake.nix to the parent directory for the same effect). The key part is that whatever path is passed in via src must include all path dependencies for the build to work

[ipetkov]

Closing as I think the issue here is Nix itself not being able to see files outside of the project directory, but feel free to reopen if you have other questions!