Using different nixpkgs channels for different machines causes excessive builds

[antifuchs]

So I just ran into an interesting problem in my "infrastructure" (basically, a remote linode box and a raspberry pi in my home): I was using two different nixpkgs release channels for the two machines, with the raspi running nixos-20.03, and the linode box running unstable. That meant that deploy-rs's nixpkgs version was pinned to the newer of the two channels. Now, when I tried deploying to the raspberry pi, it would have to download and compile a whole build toolchain (with two versions of llvm, a lot of font-rendering stuff, X libraries, gcc and gdb) - all on the Raspberry pi.

I think that's due to the "activate-rs" binary from #14 - it gets required on all target systems (which is right), but as you can't pick which toolchain it is built with, it pulls in the rustc and cargo from the toolchain on its flake inputs... and that can be quite a heavy lift.

The way I worked around this issue for now is by pinning all the deployed-to boxes to the same nixos version (20.09), which builds fast-enough on all the systems involved. But a real solution that lets me run different (and unstable) versions of nixos on subsets of the running machines would be ideal.

I'm not sure that's easily solvable (maybe by putting a activate-rs derivation into nixpkgs itself?), maybe a solution to #12 would fix this problem, too.

[antifuchs]

Ah, there is in fact a very annoying thing that requires that I use unstable nixpkgs with deploy-rs: I'm on macOS 11, which isn't supported by the rustc in nixos-20.09.

So - I think we'd need to separate the "activate-rs" binary (and associated toolchain) from the "deploy" binary (and associated toolchain).

[antifuchs]

Found a workaround: I now pin two different versions of deploy-rs. The one builds with unstable nixpkgs, to run on my local Big Sur system; the other gets used to deploy remote configurations on remote systems. This looks like this in flake.nix:

{
  inputs = {
    unstable.url = "github:nixos/nixpkgs/staging-next";
    remote-nixos.url = "github:nixos/nixpkgs/nixos-20.09";

    deploy-rs.url = "github:serokell/deploy-rs";
    deploy-rs.inputs.nixpkgs.follows = "unstable";

    deploy-rs-remote.url = "github:serokell/deploy-rs";
    deploy-rs-remote.inputs.nixpkgs.follows = "remote-nixos";
  };

  outputs = {deploy-rs, deploy-rs-remote, unstable, remote-nixos, ...}: {
    # note that this is the only place we use `deploy-rs`, built with unstable nixpkgs:
    apps = builtins.mapAttrs (_: deploy: { inherit deploy; }) deploy-rs.defaultApp;

    nixosConfigurations = {
      "monitor" = unstable.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          ./machine/monitor
          "${unstable}/nixos/modules/profiles/qemu-guest.nix"
        ];
      };
      "jumpi" = remote-nixos.lib.nixosSystem {
          system = "aarch64-linux";
          modules = [
            ./machine/jumpi.home.boinkor.net
          ];
        };
    };
    
    # everything here uses deploy-rs-remote, as it gets built & deployed with "stable" nixpkgs on the remote system:
    checks = builtins.mapAttrs (system: deployLib: deployLib.deployChecks self.deploy) deploy-rs-remote.lib;

    deploy = {
      nodes = {
        "monitor" = {
          hostname = "monitor.example.com";

          profiles.system = {
            path = deploy-rs-remote.lib.x86_64-linux.activate.nixos self.nixosConfigurations."monitor";
          };
        };

        "jumpi" = {
          hostname = "jumpi.example.com";

          profiles.system = {
            path = deploy-rs-remote.lib.aarch64-linux.activate.nixos self.nixosConfigurations."jumpi";
          };
        };
      };
    };
  };
}

[notgne2]

Could this be solved by just having the values you used out of deploy-rs-remote all take pkgs as a function? I'm not sure that's much cleaner than changing follows though

[antifuchs]

I think giving nixpkgs as an argument - or even extracting the nixpkgs to use from the nixosSystem definition if that's possible - would be great options. Not backwards-compatible, but I'd take that over having to match up deploy-rs URLs...