nixops depends on nix-channel

[bobvanderlinden]

I use NixOS on my laptop and use that laptop to deploy machine configurations to my servers. For experimentation purposes I used nixpkgs unstable channel for my laptop. Later I did a deploy in nixops and it 'upgraded' all machines to unstable versions. As you can imagine, that wasn't my intention.

There needs to be a way to get nixops to use packages independent of the system it is running on. Basically I want 'nixops deploy' to be reproducible: it shouldn't depend on system-wide repositories/options like nix-channels. Ideally, I think, this should be the default.

To override nix-channels I tried to use NIX_PATH=nixpkgs=https://nixos.org/channels/nixos-17.09/nixexprs.tar.xz nixops deploy but that resulted in:

NIX_PATH=nixpkgs=https://nixos.org/channels/nixos-17.09/nixexprs.tar.xz nixops deploy
error: hash mismatch importing path ‘/nix/store/7y6zp51ahqgn03yb7zr2pwzrjdc3274w-nixexprs.tar.xz’; expected hash ‘sha256:19b8596gq7nnzrhny7p39k52w84jdcf6jv9ddali8spb0hr7mhjz’, got ‘sha256:194z7axwwxxpxs5lmxj1ff7094jdn72qmbanqryrwac98lx0kfgl’

Another alternative, suggested by infinisil on IRC, is to use a git submodule of nixpkgs in the configuration repository where the machine configuration of nixops resides. In addition use NIX_PATH=nixpkgs=$PWD/nixpkgs nixops deploy to make sure nixops is using the right version of nixpkgs.

Even when using the URL in nixpkgs would've worked, these solutions both seem like workarounds. What is the recommended way to have nixops work independently of the local system configuration?

[aycanirican]

You can set nix.nixPath in your nixops logical configuration for each machine.

-- aycan

On 1 Oct 2017, at 02:09, Bob van der Linden [email protected] wrote:

I use NixOS on my laptop and use that laptop to deploy machine configurations to my servers. For experimentation purposes I used nixpkgs unstable channel for my laptop. Later I did a deploy in nixops and it 'upgraded' all machines to unstable versions. As you can imagine, that wasn't my intention.

There needs to be a way to get nixops to use packages independent of the system it is running on. Basically I want 'nixops deploy' to be reproducible: it shouldn't depend on system-wide repositories/options like nix-channels. Ideally, I think, this should be the default.

To override nix-channels I tried to use NIX_PATH=nixpkgs=https://nixos.org/channels/nixos-17.09/nixexprs.tar.xz nixops deploy but that resulted in:

NIX_PATH=nixpkgs=https://nixos.org/channels/nixos-17.09/nixexprs.tar.xz nixops deploy error: hash mismatch importing path ‘/nix/store/7y6zp51ahqgn03yb7zr2pwzrjdc3274w-nixexprs.tar.xz’; expected hash ‘sha256:19b8596gq7nnzrhny7p39k52w84jdcf6jv9ddali8spb0hr7mhjz’, got ‘sha256:194z7axwwxxpxs5lmxj1ff7094jdn72qmbanqryrwac98lx0kfgl’ Another alternative, suggested by infinisil on IRC, is to use a git submodule of nixpkgs in the configuration repository where the machine configuration of nixops resides. In addition use NIX_PATH=nixpkgs=$PWD/nixpkgs nixops deploy to make sure nixops is using the right version of nixpkgs.

These solutions both seem like workarounds. What is the recommended way to have nixops work independently of the local system configuration?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[AmineChikhaoui]

Hi,

If you'd like a reproducible deployment then you'll likely need a local checkout of nixpkgs-channels (to make sure you get pkgs from the binary cache) then you can set that by doing nixops modify -d <deployment_name> <nix_expr> -I nixpkgs=/path/to/nixpkgs-channels.

On Sun, Oct 1, 2017 at 1:09 AM, Bob van der Linden <[email protected]

wrote:

I use NixOS on my laptop and use that laptop to deploy machine configurations to my servers. For experimentation purposes I used nixpkgs unstable channel for my laptop. Later I did a deploy in nixops and it 'upgraded' all machines to unstable versions. As you can imagine, that wasn't my intention.

There needs to be a way to get nixops to use packages independent of the system it is running on. Basically I want 'nixops deploy' to be reproducible: it shouldn't depend on system-wide repositories/options like nix-channels. Ideally, I think, this should be the default.

To override nix-channels I tried to use NIX_PATH=nixpkgs=https:// nixos.org/channels/nixos-17.09/nixexprs.tar.xz nixops deploy but that resulted in:

NIX_PATH=nixpkgs=https://nixos.org/channels/nixos-17.09/nixexprs.tar.xz nixops deploy error: hash mismatch importing path ‘/nix/store/7y6zp51ahqgn03yb7zr2pwzrjdc3274w-nixexprs.tar.xz’; expected hash ‘sha256:19b8596gq7nnzrhny7p39k52w84jdcf6jv9ddali8spb0hr7mhjz’, got ‘sha256:194z7axwwxxpxs5lmxj1ff7094jdn72qmbanqryrwac98lx0kfgl’

Another alternative, suggested by infinisil on IRC, is to use a git submodule of nixpkgs in the configuration repository where the machine configuration of nixops resides. In addition use NIX_PATH=nixpkgs=$PWD/nixpkgs nixops deploy to make sure nixops is using the right version of nixpkgs.

These solutions both seem like workarounds. What is the recommended way to have nixops work independently of the local system configuration?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/NixOS/nixops/issues/736, or mute the thread https://github.com/notifications/unsubscribe-auth/AE6SwUm5wKJgsUL3415OWVTro1Gw5CvIks5snsopgaJpZM4PpuRx .

-- Amine

[bobvanderlinden]

Yes, on IRC it was suggested to add a git submodule in the repo where my nixops configuration resides. It indeed seems like an alright solution.

My main concern is that this is something that should be default. It's very dangerous that nixops deploy depends on system-wide nixpkgs without that being very explicit in the command-line or documentation. I ran into this by accident updating my server to nixpkgs-unstable and I can imagine others doing the same thing. In addition, the server only being able to update when the system is updated.

It 'feels' dirty. It is doing the exact thing NixOS is trying to avoid: depend on system-wide and implicit settings. Nixops should have its own channel/update procedure.

Maybe this ties into https://github.com/NixOS/nix/issues/779 ? Which suggests removing nix-channel altogether and relying on URLs in NIX_PATH.

EDIT: I do like @aycanirican suggestion of using nix.nixPath. Combining that with an URL does seem to resolve the issue I was having:

    nix.nixPath = [
      "nixpkgs=https://nixos.org/channels/nixos-17.09/nixexprs.tar.xz"
    ];

Should this be documented in the nixops manual?

[aycanirican]

You may want to use a shell.nix such that:

let pkgsSrc = builtins.fetchTarball { url = "https://github.com/NixOS/nixpkgs-channels/archive/ad6265dbfae1279302b689425eec57eab7b5948b.tar.gz"; };
     pkgs = (import pkgsSrc {});
     nixopsSrc = builtins.fetchTarball https://github.com/NixOS/nixops/archive/ca9a942ea9da7102d38e1718807460e853cdf867.tar.gz;
     nixops-latest = (import "${nixopsSrc}/release.nix" {});
in  pkgs.stdenv.mkDerivation rec {
  name = "nixops-env";
  buildInputs = with pkgs; [ openssl awscli nixops-latest.build.x86_64-linux ];
  shellHook = ''
    . ~/.aws/vars
    export NIX_PATH=${pkgs.path}:nixpkgs=${pkgs.path}:.
  '';
}

Which ensures that a specific channel hash and a specific nixops exists in the environment.

Cheers, — aycan

On 2 Oct 2017, at 23:40, Bob van der Linden [email protected] wrote:

Yes, on IRC it was suggested to add a git submodule in the repo where my system configuration resides. It indeed seems like an alright solution.

My main concern is that this is something that should be default. It's very dangerous that nixops deploy depends on system-wide nixpkgs without that being very explicit in the command-line or documentation. I ran into this by accident updating my server to nixpkgs-unstable and I can imagine others doing the same thing. In addition, the server only being able to update when the system is updated.

It 'feels' dirty. It is doing the exact thing NixOS is trying to avoid: it depends on system-wide and implicit settings. Nixops should have its own channel/update procedure.

Maybe this ties into NixOS/nix#779 ? Which suggests removing nix-channel altogether and relying on URLs in NIX_PATH.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[bobvanderlinden]

The previously mentioned option nix.nixPath = [ ... ] does not work as intended. It does not seem to be used while building the machine configuration. (does not seem to download anything when using a nix.nixPath = [ "nixpkgs=URL" ].

Using nix-shell is indeed a good option. I gave it a go, altered your example a bit and it seems be workable: https://gist.github.com/bobvanderlinden/e9ef4c338882f1ef31424e4743be15fc

That being said, should this be documented for nixops? The current documentation doesn't state anything on nixops depending on the running systems configuration/channels. In hindsight I see this could have been obvious, but it seems dangerous when nixops is being used in production.

[eqyiel]

Another thing you can do is use something like direnv and set a default NIX_PATH in your deployments repo: https://github.com/eqyiel/deployments/blob/master/.envrc#L4

[bobvanderlinden]

@eqyiel That's a fine solution when working by myself. I'm more interested in a documented/standard/recommended workflow that can work for teams as well.

When I'm showing people nixops it doesn't motivate them to get started when they see the 'extra baggage' that is needed to get a reproducible environment. Whether that 'extra baggage' is direnv, nix-shell, NIX_PATH does not really matter. I do have to say that nixops has a more stable environment than most other deploytools have, but reproducability is Nix's main selling point and nixops doesn't really adhere to that point without the 'extra baggage'. Especially when that 'extra baggage' isn't really documented.

[eqyiel]

@bobvanderlinden FWIW, that is what I do with my team in our private repo and they are able to understand that they need to install direnv from a "getting started" note in the README.

NIX_PATH is not extra baggage, that's just how you tell Nix where to find nixpkgs (and expressions for nixops) if you don't want to use -I nixpkgs=/path/to/nixpkgs everywhere. It's documented in the Nix manual: https://nixos.org/nix/manual/#sec-common-env

How do you suggest this be made clearer?

I think another problem here is that nix-channel is confusing, even for experienced users:

https://github.com/NixOS/nixpkgs/issues/7113 https://github.com/NixOS/nix/issues/813 https://github.com/NixOS/nix/issues/1548

[bobvanderlinden]

Yes, I absolutely agree nix-channel is confusing. It might also be the root of this issue. From what I understand nix-channel will be deprecated sooner or later (probably later ;)).

That said, it's still confusing that the documentation doesn't warn anyone using nixops deploy to make sure the same nixpkgs is used every time it is executed.

If I were to use nixops in a team I would make sure to don't tell anyone to use nixops by themselves, but create a shell script that uses any of the above suggestions to run nixops deploy.

The problem is that if anyone runs nixops deploy without any of the above suggestions it would deploy a very different version to a whole cluster!

Having nix_path defined (and used) in the machine/cluster configuration seems like the best solution. Currently it isn't picked up by nixops it seems.

[teto]

[temporary hijack since it seems related] Is it possible that nixops affect nixos-rebuild in some way ? like retaining references to some package that make this package reused by the system instead of a newer version. While updating the strongswan package I've sometimes noticed network manager reusing the old strongswan after a nixos-rebuild even though it had built the new one.

[bobvanderlinden]

I don't know how nixops itself could affect nixos-rebuild. I can imagine doing a nix-channel update, because you want to do an update through nixops, will also affect the next run of nixos-rebuild.

[wmertens]

the nix.nixPath = [ "nixpkgs=https://nixos.org/channels/nixos-19.03/nixexprs.tar.xz" ]; thing works, but you also need it on your deploy host.

[asymmetric]

On the deploy host I use a nix shell to pin NIX_PATH.

[sephii]

I feel like the nix shell solution is also a workaround rather than a proper solution to this problem. i’m using nixops to deploy to different networks, which could be running different NixOS versions. This means even with a nix shell I could by mistake run nixops deploy on a network that’s supposed to be deployed from a different nix shell with a different pin.

I would love to be able to set the nixpkgs from inside the nix expression for the network.

[bjornfor]

I would love to be able to set the nixpkgs from inside the nix expression for the network.

I think that's working as of https://github.com/NixOS/nixops/pull/1422. (But it's not part of any release yet.)