Use cachix-action on a NixOS host

[peperunas]

What is the best way to set-up cachix on a NixOS host running with a services.github-runner?

As per the issue raised on the NixOS matrix channel, this is a log I got from my attempt.

 /nix/store/l0wlqpbsvh1pgvhcdhw7qkka3d31si7k-bash-5.1-p8/bin/bash -c nix-env --quiet -j8 -iA cachix -f https://cachix.org/api/v1/install
  error: opening lock file '/nix/var/nix/profiles/per-user/github-runner/profile.lock': Read-only file system
  Error: Action failed with error: Error: The process '/nix/store/l0wlqpbsvh1pgvhcdhw7qkka3d31si7k-bash-5.1-p8/bin/bash' failed with exit code 1

[domenkozar]

How exactly are you running github runner?

[peperunas]

May I ask what do you mean with this?

This is the .yml file:

  build-release:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v2
      - uses: cachix/cachix-action@v10
        with:
          name: mycache
          authToken: "${{ secrets.CACHIX_AUTH_TOKEN }}"
      - run: nix build -L

The runner is simply enabled in my configuration.nix, nothing fancy.

[domenkozar]

Could you also paste configuration.nix snippet so I can reproduce?

[peperunas]

Sure!

Here it is:

{
  services.github-runner = {
    enable = true;
    url = "https://github.com/project";
    tokenFile = "/secrets/github-runner/token";
    replace = true;
  };

  nix.extraOptions = ''
    tarball-ttl = 0
    access-tokens = github.com=token
  '';
}

[domenkozar]

It seems like the user doesn't have access to the profile, I can take a look at this next week.

[peperunas]

Is there anything I can do to help? I could try to give it a go as it may be a good learning opportunity for me :-)

[domenkozar]

You can check who's the owner of /nix/var/nix/profiles/per-user/github-runner and under what user does the cachix-install commands run under

[peperunas]

So, I took a look a few days ago but then I got sidetracked. It seems that everything is owned by github-runner, I am not sure what's happening

[domenkozar]

I'll take a look tomorrow!

[peperunas]

Sorry, actually the builds are running under nixbld and the profile is owned by github-runner, my bad, sorry Domen!

[miuirussia]

I have same error, @peperunas please tell me how you fixed it?

[peperunas]

Unfortunately, I didn’t solve it. On 20 Feb 2022, 20:12 +0000, Kirill Kuznetsov @.***>, wrote:

@peperunas please tell me how you fixed it? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[miuirussia]

I solved it with my cachix-action fork, and adding cachix to services.github-runner.extraPackages

@peperunas @domenkozar

[domenkozar]

Happy to accept a PR to skip installation if cachix is already installed!

[miuirussia]

Ok, done: #109

[steveej]

i'm facing this as well and i'm noticing that even though cachix is in the github-runner's extraPackages, it doesn't show up in the PATH within cachix-action's scripts.

so the following still tries to install cachix (and fails to do so):

      - name: Setup cachix
        uses: cachix/cachix-action@v10
        with:
          (...)
          installCommand: |
            if ! type -f cachix; then
              nix-env -if https://github.com/cachix/cachix/tarball/master \
                --substituters 'https://cache.nixos.org https://cachix.cachix.org' \
                --trusted-public-keys 'cachix.cachix.org-1:eWNHQldwUO7G2VkjpnjDbWwy4KQ/HNxht7H4SSoMckM= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY='
            fi

i would expect the above to work. that's because i verified in a separate step that cachix is indeed available:

      - name: Check if cachix is installed
        id: cachix_exists
        run: |
          cachix --version
        continue-on-error: false

here's some proof from a slightly modified version of the above where i modified the install script to rely on the external existence check:

[domenkozar]

I'll try to get #109 merged to address this.

[domenkozar]

@steveeJ could you try using https://github.com/cachix/cachix-action/pull/123 branch?

[domenkozar]

Released v11 that addresses the issues here, please let me know if it works so we can close :)

[steveej]

Released v11 that addresses the issues here, please let me know if it works so we can close :)

i just tried and the installation of cachix, or rather skipping thereof in my case, works!

my remaining issue is in getting the cachix settings to actually take effect. this is from an SSH session with the runner's context of this CI job instance:

[github-runner@nixos:~/holochain/holochain]$ cat /etc/nix/nix.conf
# WARNING: this file is generated from the nix.* options in
# your NixOS configuration, typically
# /etc/nixos/configuration.nix.  Do not edit it!
allowed-users = *
auto-optimise-store = false
builders =
cores = 0
experimental-features = nix-command flakes
extra-sandbox-paths =
max-jobs = auto
require-sigs = true
sandbox = true
sandbox-fallback = false
substituters = https://cache.nixos.org/
system-features = nixos-test benchmark big-parallel kvm
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
trusted-substituters =
trusted-users = root github-runner sshsession


[github-runner@nixos:~/holochain/holochain]$ cat ~/.config/nix/nix.conf
substituters = https://cache.nixos.org https://cache.nixos.org/ https://holochain-ci.cachix.org
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= holochain-ci.cachix.org-1:5IUSkZc0aoRS53rfkvH9Kid40NpyjwCMCzwRTXy+QN8=

[github-runner@nixos:~/holochain/holochain]$ id
uid=61779(github-runner) gid=61779(github-runner) groups=61779(github-runner)

[github-runner@nixos:~/holochain/holochain]$ nix-shell https://holochain.love
warning: ignoring untrusted substituter 'https://holochain-ci.cachix.org'

i'm not sure where this global /etc/nix/nix.conf actually lives. the ones on the nixos host and the nixos-container instance that runs the github-runner.service looks different, and has only trusted-users = root set. this is probably the issue that renders the cachix settings ineffective in my case.

[steveej]

i'm not sure where this global /etc/nix/nix.conf actually lives. the ones on the nixos host and the nixos-container instance that runs the github-runner.service looks different, and has only trusted-users = root set. this is probably the issue that renders the cachix settings ineffective in my case.

i found what i consider a workaround to the issue i've explained.

in my situation there's a nixos host that runs multiple nixos containers which in turn run github-runner. to get the "github-runner" user trusted to effectively get cachix-action to work from a workflow, i added this to the host's and the container's nixos config:

  nix.settings.trusted-users = [
    "root"
    "github-runner"
  ];
  users.users.github-runner = {
    uid = 1000;
    isSystemUser = true;
    createHome = false;
    group = "github-runner";
  };
  users.groups.github-runner = {};

only if i add this in both places is the host's nix-daemon happy with allowing the workflow to introduce ad-hoc extra-substituters

[domenkozar]

Thanks! I'm closing this as it seems it's possible to use cachix-action on a NixOS host. Please reopen if any issues persist.