Running headscale commands as non-root produces generic "deadline exceeded" error message

[victorhooi]

Bug description

If you attempt to run "headscale" commands as a non-root user, it prints out a generic and somewhat unhelpful "context deadline exceeded" error message:

victorhooi@headscale-test:~$ headscale nodes list
2022-03-23T08:56:46Z FTL ../runner/work/headscale/headscale/cmd/headscale/cli/utils.go:482 > Could not connect: context deadline exceeded error="context deadline exceeded"

This is rather unhelpful, and doesn't really tell the user where the problem is.

I upped the logging level to "trace", and noticed that it was trying to access /var/run/headscale.sock - I re-ran with sudo, and the command worked:

victorhooi@headscale-test:~$ sudo headscale nodes list
ID | Name | NodeKey | Namespace | IP addresses | Ephemeral | Last seen | Online | Expired

Ideally:

1. It should be documented somewhere (e.g. README) that all headscale commands, including things like listing nodes need to run as the root user
2. headscale should detect when run as a non-root user, and print out a friendlier error message. (e.g. `headscale requires permissions to run. Please run as root")
3. Potentially if there's some way of running headscale as non-root - e.g. by adding instructions to README on setting up a *nix group with the right permissions needed etc. - or maybe changing permissions on the headscale.sock file - this should be documented.

To Reproduce

Run a headscale command as a normal (non-root) user.

Context info

[e-zk]

Hello! headscale runs as a non-root user and seems to work fine in my setup.

I think what's wrong here could be your file permissions on the headscale related stuff in /var/*. Could you run a quick ls -l on those files to make sure that isn't the case?

ls -l /var/run/headscale.sock

Edit: I see you've actually mentioned this in your post - adding docs about how to change these permissions seems like a good idea. I'll see what I can do.

[victorhooi]

Here are the permissions for the socket file on my system:

$ ls -l /var/run/headscale.sock
srwxrwx--- 1 root root 0 Mar 23 16:46 /var/run/headscale.sock

This is on a Debian 11 machine.

From memory, when I was following along the docs at https://github.com/juanfont/headscale/blob/main/docs/running-headscale-linux.md - I had to use sudo to run the mkdir commands:

sudo mkdir -p /var/lib/headscale

(I assume that's expected behaviour, right?)

Or is there another way we should create those directories, with user access?

[e-zk]

Yeah so I think that's the root (heh) of the problem.

When you've created that directory with sudo it's given ownership rights to root and root only.

If you're running headscale as a non-root user you should change the ownership of this dir to that non-root user.

Try running:

# chown -R hscale:hscale /var/lib/headscale

Where hscale is the username of your non-root user headscale is running as.

[victorhooi]

Hmm - ok, so you're saying we should create a dedicated headscale (or hscale) user and group, and set ownership of the headscale directories (/etc/headscale, /var/lib/headscale to that.). Should the /usr/local/bin/headscale binary also be set to belong to headscale:headscale?

If this is the recommended approach - we should document it in https://github.com/juanfont/headscale/blob/main/docs/running-headscale-linux.md.

There is mention of a headscale user, but it's a passing reference at the bottom of the page in the systemd section. I suspect many people are like me, and just start at the "Configure and run headscale" section at the top 😛, and follow the instructions.

[e-zk]

I'm not saying you should create a dedicated headscale user (unless you want to - I personally do).

You said you are running headscale as a non-root user, correct? What I am saying, is to make sure that non-root user has the right permissions to access the config files and stuff in /var.

[e-zk]

Because that non-root user currently can't access anything in /var, because it's all owned by root. You understand that, right?

[victorhooi]

Yup, I do understand it =) - thanks for clarifying.

My question is more - if somebody is just blindly following the instructions at https://github.com/juanfont/headscale/blob/main/docs/running-headscale-linux.md - they're going to need to run headscale as root, which sounds like it's not necessarily a great idea.

Would it make sense to add some steps there - possibly just add a headscale user, and set appropriate ownerships on the folders/files?

[thenitai]

hmmmm........ I came here because of that "context deadline exceeded" issue. However in my experience it's because I'm running headscale with systemd.

According to the docs you would need to change the path for the sock. However, when you do that the command "headscale namespaces list" throws that error.

As soon as you stop the deamon. Change the socks path and start with "headscale server" all commands work again.

[thenitai]

Ok, I got it to work. All that is required is to create the directory for the sock:

mkdir -p /var/run/headscale

and then it works with the path to the socket as described in the docs, i.e., unix_socket: /var/run/headscale/headscale.sock