`workspaceShell`: "Argument list too long"

[mainrs]

When using workspaceShell I get this error when running nix develop on the flake below:

$ nix develop
error: builder for '/nix/store/hnj8smywwcvynfrvll603ssmzzn99lzw-nix-shell-env.drv' failed with exit code 1;
       last 2 log lines:
       >
       > ErroSysError{executing '/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash': Argument list too long
       For full logs, run 'nix log /nix/store/hnj8smywwcvynfrvll603ssmzzn99lzw-nix-shell-env.drv'.

$ nix log /nix/store/hnj8smywwcvynfrvll603ssmzzn99lzw-nix-shell-env.drv
^A
^E^@^@^@^@^@^@^@Error^@^@^@^@^@^@^@^@^@^@^@^H^@^@^@^@^@^@^@SysError{^@^@^@^@^@^@^@executing '/nix/store/4nmqxajzaf60yjribkgvj5j54x9yvr1r-bash-5.1-p12/bin/bash': Argument list too long^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

{
  inputs = {
    cargo2nix.url = "github:cargo2nix/cargo2nix";
    flake-compat = {
      url = "github:edolstra/flake-compat";
      flake = false;
    };
    flake-utils.url = "github:numtide/flake-utils";
    nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
    rust-overlay.url = "github:oxalica/rust-overlay";
    rust-overlay.inputs.nixpkgs.follows = "nixpkgs";
    rust-overlay.inputs.flake-utils.follows = "flake-utils";
  };
  outputs = { self, cargo2nix, flake-utils, nixpkgs, rust-overlay, ... }:
    flake-utils.lib.eachDefaultSystem (system: 
      let 
        overlays = [
          (import "${cargo2nix}/overlay")
          rust-overlay.overlay
        ];
        pkgs = import nixpkgs {
          inherit overlays system;
        };

        rustPkgs = pkgs.rustBuilder.makePackageSet' {
          rustChannel = "1.59.0";
          packageFun = import ./Cargo.nix;
        };

        workspaceShell = rustPkgs.workspaceShell {};
      in
      {
        devShell = workspaceShell;

        packages = {
          app = (rustPkgs.workspace.app {}).bin;
        };
      }
    );
}

# Cargo.toml
[build-dependencies]
tauri-build = { version = "1.0.0-rc.4" }

[dependencies]
polars = "0.20"

serde_json = "1.0"
serde = { version = "1.0", features = ["derive"] }
tauri = { version = "1.0.0-rc.4", features = [ "api-all" ] }

[graysonhead]

I'm getting the exact same error on one of my projects as well:

https://github.com/graysonhead/jager/blob/cargo2nix_refactor/flake.nix

[grayson@gdesktop jager]$ nix develop
error: builder for '/nix/store/6qa0j4l11rmxyzll1syz3i1rh3yvyakq-nix-shell-env.drv' failed with exit code 1;
       last 2 log lines:
       > 
       > ErroSysError{executing '/nix/store/fcd0m68c331j7nkdxvnnpb8ggwsaiqac-bash-5.1-p16/bin/bash': Argument list too long
       For full logs, run 'nix log /nix/store/6qa0j4l11rmxyzll1syz3i1rh3yvyakq-nix-shell-env.drv'.

[psionic-k]

I believe this will happen in the shell first because all of the arguments wind up getting merged into one list and applied to the environment, likely without any de-duplication. There was a similar issue within single dependencies earlier. For all of the dependencies and state required to set up the shell, it will just happen sooner.

[psionic-k]

Does this still affect master? Be sure to check release notes and have a look at the new flake examples.

[psionic-k]

Still affects master. This is the uniq count for propagatedBuildInputs from nix show-derivation <.drv> for the devShell

   2709 /nix/store/4i79qzjn3y3k1wmyigwcrpi83xhj3cxx-pkg-config-propagate-env
   2709 /nix/store/561d5sd2r3ih63v7n38k2pgx4bhibl0m-pkg-config-wrapper-0.29.2
     84 /nix/store/5wj9igmv1x7lsdzibcy6mpxm6p9r5p91-pango-1.48.10-dev
   1414 /nix/store/9hvamg7jyhmcww590z2g0gy1f8rkil6m-glib-2.70.1-dev
      3 /nix/store/dd2nqqzicf6ddmh9afs2laq8x2ci7msl-openssl-sys-propagate-env
      1 /nix/store/f1pvni01z4h6qsykl4vgf6v7mknm73r6-dbus-1.12.20-dev
      7 /nix/store/fbf06z4wqwdcampr8fjwkr838gijcgpg-libsoup-2.74.1-dev
     85 /nix/store/hwgssqmwfd7kprnga858ah0rfh5fagpw-gdk-pixbuf-2.42.6-dev
     41 /nix/store/kgc194paj8z8hjsxdzmi753ppg2v8642-gtk+3-3.24.30-dev
     30 /nix/store/s6r2s26dyr287691fbx9bq6hg315qm8n-atk-2.36.0-dev
     88 /nix/store/vhwcgzyhzvvnhga6gm457fqdvm0ggiph-cairo-1.16.0-dev
      7 /nix/store/yn5y39dii8rwqg8azgxlz9f6158r9j1h-webkitgtk-2.36.1-dev

[psionic-k]

Found the cause.

The dev shell support in workspaceShell is mostly pass-through to mkShell https://github.com/cargo2nix/cargo2nix/blob/master/overlay/workspace-shell.nix#L8-L16

In order to shove environment variables all the way through to the development shell, we're using propagatedBuildInputs https://github.com/cargo2nix/cargo2nix/blob/master/overlay/overrides.nix#L47-L62

A different mechanism for sending the environment information is necessary. I don't really like creating these dummy derivations and abusing propagatedBuildInputs if we can ourselves process inputsFrom, reading some attribute from the derivations while traversing the whole dependency tree. If we do that, we can pick up these environment variables without duplication and without this slightly crazy mechanism that's giving us 500kb shell variables in this case.

[psionic-k]

nix show-derivation github:cargo2nix/cargo2nix shows the duplication in propagatedBuildInputs. I'm going to have to find a new mechanism to get the information for mkShell without creating new derivations.

Probably writing a new override shortcut (#251). What needs to happen is to both add the shell variable to the derivation and to append some extra data that can be extracted later and passed to mkShell without duplication. The development shell is it's own derivation, and this shouldn't affect builds nor abuse propagatedBuildInputs.

[psionic-k]

The workaround is to use the overrideAttrs of the shell (it's just pkgs.mkShell with some conveniences added). You can completely clear out the list of propagatedBuidInputs. If it turns out you needed the duplicated values, de-duplicate them and copy them in to the override. I'm working on a better fix.

        workspaceShell = (rustPkgs.workspaceShell {
          shellHook = ''
            export PS1="\033[0;31m☠dev-shell☠ $ \033[0m"
          '';
        }).overrideAttrs (finalAttrs: prevAttrs: {
          shellHook = ''
            export PS1="workaround $"
          '';
          propagatedBuildInputs = [ ];
        });

[psionic-k]

Please update to the unstable branch if you still have this issue. You can view the excessively duplicated propagateEnv derivations by showing the shell:

nix show-derivation .#devShells.x86_64-linux.default in cargo2nix for example

The massive environment duplication appears to have subsided, although I still want to fix the abuse of propagatedBuildInputs as a mechanism of propagatting environment variables from dependencies all the way to the dev shell.

[psionic-k]

Fix is available in #292

I made an overhaul of the workspace shell.

The job is actually fairly simple. Workspace shell makes cargo able to build the rust DAG by itself. This means setting up shell variables and providing paths that are not Rust deps and not inside the rust DAG. See the code in https://github.com/cargo2nix/cargo2nix/pull/292/commits/7bdff4b2fbc21be166790805faa708cdf15bf2ea for details.

Note that many overrides have been changed to use the correct dependency sets now that workspace shell can express those side effects without abuse of propagation.

unstable branch will contain this fix soon and it will go into a release branch at 0.12. I'm not planning to put any more commits on to 0.11

[mainrs]

Thank you so much! I am thankful by how much effort you put into the tool!