Cache nix-shell

[arximboldi]

Hi!

The way my Github Actions are set up, I install Nix and then use nix-shell. I was wondering if there is a way to use this action such that instead of using a specific packages.nix file it caches whatever is installed directly or indirectly by nix-shell using shell.nix?

[tbenst]

I tried a workaround, but still am not getting nix-shell to cache:

I have a release.nix file that builds both a shell and my package.

shell.nix is now trivial: (import ./release.nix { }).shell

As is default.nix: (import ./release.nix { }).project

Here's my workflow (file here):

- name: Cache install Nix packages
    uses: nix-actions/[email protected]
    with: # update key to invalidate cache
    key: nix-v2-${{ hashFiles('release.nix') }}
    nix_file: 'release.nix'
    env:
    ACTIONS_ALLOW_UNSECURE_COMMANDS: 'true'
- run: nix-shell shell.nix --run "echo OK"

For some reason, the nix-shell step has to re-download everything into nix store.

Any ideas what is going on?

[rikhuijzer]

Sorry, I actually have no idea. I've hacked together this project in a few days and am now unable to recall the details for fixing those issues. Maybe https://github.com/dbaynard/cache-nix-store @dbaynard can help you out. He currently knows all the details, but unfortunately his tests seem to be failing

[rikhuijzer]

For some reason, I know again. I just had to sleep on it, apparently. @tbenst your workaround probably isn't working because the cache is uploading during the uses: nix-actions/[email protected] (this should now be uses: rikhuijzer/[email protected]) step. The nix-shell is the next step and is not uploaded into the cache indeed. I've also tried to run the upload at the end of the GitHub workflow, similarly to the GitHub cache Action. Unfortunately, that wasn't working out because then I had to use a composite action which wasn't working with other features. All in all, this project was one big trade-off, because the GitHub Actions aren't/weren't very flexible.

@arximboldi I've considered supporting both a nix-shell and doing a global installation when making this package. For me, the global installation was more convenient because otherwise you need to specify nix-shell shell.nix --run in every step. In GitHub runners, there wasn't a nice way to override this that I could think of.

Instead, what you can do is install all the packages globally via this action and then call a shell with the same dependencies. Then, NixOS doesn't have to download all the dependencies again. I think that this should be possible by encapsulating the nix files in a shell script (something with import).

[dbaynard]

He currently knows all the details

This is a very kind interpretation :laughing: Though I see you fixed #3, so my branch is no longer needed (I hope it was helpful).

I hadn't completely understood what the action was doing: I think now that #3 is fixed I'll leave this and focus on getting the cache to work in run steps (I've been using the cachix action to install nix, so that's not needed).

On using nix: I've just installed things globally in the past — I recommend nix run -L (-L for logging), which avoids the need to quote the resulting expression. I couldn't get logging to work properly, though, so perhaps I should use nix-shell --run myself.

[diamondburned]

I've considered supporting both a nix-shell and doing a global installation when making this package. For me, the global installation was more convenient because otherwise you need to specify nix-shell shell.nix --run in every step. In GitHub runners, there wasn't a nice way to override this that I could think of.

Is it possible to:

1. Run nix-instantiate -E '(import ./shell.nix {}).buildInputs' to get the derivations for the packages, and
2. Run nix eval --raw '(import ./shell.nix {}).shellHook' after initializing dependencies?

We can ensure that the derivations are present by doing nix-shell --run true.

[haizaar]

@diamondburned Can you please elaborate how I can supply packages to install for this action from shell.nix file? I.e. what should I write in mypackages.nix for it to advertize packages from shell.nix's buildInputs ?

[diamondburned]

Can you please elaborate how I can supply packages to install for this action from shell.nix file?

You can't. This issue is discussing how this should be implemented.

There's a workaround that you can do, which is to have a file listing packages as a function of pkgs: [], then one file for cache-install to use and one for shell.nix to use. Kind of like this workflow.

[haizaar]

Thanks! That's what I was looking for. However caching doesn't seem to work. Here are my files:

$ cat .nix/action-deps.nix 
let commit = "ce6aa13369b667ac2542593170993504932eb836"; # nixos-22.05 official
  pkgsSrc = builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/${commit}.tar.gz";
    # Use "nix-prefetch-url --unpack <url>" to calculate sha256
    sha256 = "0d643wp3l77hv2pmg2fi7vyxn4rwy0iyr8djcw1h5x72315ck9ik";
  };
  pkgs = import pkgsSrc {};

in import ./deps.nix {
  inherit pkgs;
}

$ cat .nix/deps.nix 
{ pkgs }:

with pkgs; [
  terraform
  google-cloud-sdk
  envsubst
]

$ cat shell.nix 
{ pkgs ? import <nixpkgs> {} }:

let actionDeps = import ./.nix/action-deps.nix;

in pkgs.mkShell {
  buildInputs = actionDeps;
}

My GH workflow is:

...
    - uses: actions/checkout@v3

    - name: Install Nix packages
      uses: rikhuijzer/[email protected]
      with:
        key: nix-${{ hashFiles('.nix/**') }}
        nix_file: ".nix/action-deps.nix"

    - name: Init Nix shell
      run: nix-shell --run true

While the cache is loaded during the run, it's not used by the nix-shell --run true command - it downloads everything every time. Any ideas?

[diamondburned]

nix-shell --run true uses a different Nixpkgs than the one written in the Nix file.

You shouldn't need nix-shell at all. cache-install installs packages to the global environment.

Edit: I'm reading the file again and it seems like it's using the same Nixpkgs for the buildInputs. Perhaps the stdenv is different which also changes the packages' hashes?

[haizaar]

Oh, I get it, I should just run commands as if I was already in the nix-shell (like I am in my local dev). That works indeed and works well.

I'm still curious what triggers nix-shell to pull the packages again. May it be the case that the action sets nixpkgs to the tip of the channel or something, so if my sha256 is not the latest, it's considered out of date? (sorry if talking non-sense, my nix skills are quite basic still). One hint to that, is that when I switch to this separated config with tarball hash in .nix/action-deps.nix then my local shell refeteched bunch of stuff (and I'm on NixOS 22.05 as well locally ) Otherwise, the env looks the same.

[diamondburned]

I have found a way to install nix-shell packages:

nix-env -iE '{ ... }: (import ./shell.nix {}).buildInputs'

You can also directly feed this file into nix-build to get the output derivation path:

nix-build --out-link /tmp/cache-output ./shell.nix

[diamondburned]

I've made https://github.com/rikhuijzer/cache-install/pull/12 to implement this.

[diamondburned]

I'm also making a soft fork of cache-install to experiment with some cool features, such as using the Nix output hashes as the cache key: https://github.com/diamondburned/cache-install/tree/auto-cache.

I'm currently using it like so:

      - name: Install Nix packages
        id: nix-install
        uses: diamondburned/cache-install@57ae7b74633e956428c92f69f793bcf654149998
        with:
          auto-optimise: true
          shell-file: shell.nix
          instantiated-files: servers/*/default.nix

This allows me to include both the shell.nix file and all of my server deployment files as the input for the cache hash automatically.

[haizaar]

@diamondburned curious what's the status of your work and if there is a chance it will make it into upstream.

[diamondburned]

@diamondburned curious what's the status of your work

I've been using it for a few months now, and it's been working perfectly fine. Currently, all you really need is:

- uses: diamondburned/cache-install@main

and if there is a chance it will make it into upstream

It won't. See https://github.com/rikhuijzer/cache-install/pull/12.

If you're using Flakes, it might also be worth looking at magic-nix-cache-action.