Unable to parse newly created API key from Headscale

[larrydewey]

trafficstars

Overview:

When attempting to connect to a newly instantiated instance of headscale, I am attempting to generate a new API key, as per the instructions from the /admin interface. However, when I paste the newly created API key, I am getting an error saying it is unable to be parsed. I have tried using older versions, as well, but that does not seem to fix the issue.

docker-compose.yaml

version: '3.8'
services:
  headscale:
    image: 'headscale/headscale:0.23.0-alpha11'
    container_name: 'headscale'
    restart: 'unless-stopped'
    command: 'serve'
    volumes:
      - './data:/var/lib/headscale'
      - './config:/etc/headscale'
    ports:
      - '8081:8080'
      - '9090:9090'
      - '50443:50443'
    environment:
      TZ: 'America/Chicago'

  headplane:
    container_name: headplane
    image: ghcr.io/tale/headplane:latest
    restart: unless-stopped
    volumes:
      - './data:/var/lib/headscale'
      - './config:/etc/headscale'
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
    ports:
      - '8888:3000'
    environment:
      HEADSCALE_URL: 'http://headscale:8080'
      COOKIE_SECRET: '<redacted>'
      API_KEY: '<redacted>'


      # These are the default values
      HOST: '0.0.0.0'
      PORT: '3000'

Generating API key from Headscale:

ash-4.4# docker-compose exec -it headscale headscale apikeys create
2024-05-27T09:55:17-05:00 TRC DNS configuration loaded dns_config={"Nameservers":["9.9.9.9"],"Proxied":true,"Resolvers":[{"Addr":"9.9.9.9"}]}
An updated version of Headscale has been found (0.23.0-alpha9 vs. your current v0.23.0-alpha11). Check it out https://github.com/juanfont/headscale/releases
m92-EDp_-w.ybA6119_PajJHvdZsrw9m6aGAlEdiOVJopDkeIni9w8

Attempting to use the API Key

docker-compose logs from headscale when attempting to use the API key

headscale  | 2024-05-27T09:56:11-05:00 ERR home/runner/work/headscale/headscale/hscontrol/app.go:398 > failed to validate token error="failed to parse ApiKey" client_address=172.22.0.3:39080
headplane  | HeadscaleError: Unauthorized
headplane  |     at pull (file:///app/build/server/index.js?t=1716401776000:896:11)
headplane  | GET /admin/machines?_data=routes%2F_data.machines._index 500 - - 10.767 ms
headplane  | GET /admin/login?_data=routes%2Flogin 200 - - 5.428 ms

[tale]

Are you running Headscale and Headplane on different domains? There could be possible CORS issues. In general the handling for that is pretty brittle and I'd ideally want to fix it anyways.

[larrydewey]

They are both being deployed via docker compose on the same node. The domain will be the same for accessing both, just a different port.

[tale]

I'm under the impression that it is actually a CORS error that isn't being propagated into logs or the user UI for some reason, the API key does work if you query the API yourself with cURL right? curl -H 'Authorization: Bearer <token>' https://myheadscaleinstance.com/api/

[mitchellkellett]

I'm also having this issue, I created a separate Headscale instance so I could test Headplane without breaking my main instance.

Both Headscale and Headplane are running on the same node via docker compose. It's proxied by NPM in the same way that my main instance of Headscale is.

I can query the API myself using curl -H 'Authorization: Bearer *****' http://myheadscaleinstance.com/api/v1/apikeys from both my device and from inside the Headplane container.

headscale   | 2024-06-15T03:48:08Z ERR home/runner/work/headscale/headscale/hscontrol/app.go:414 > failed to validate token error="failed to parse ApiKey" client_address=10.0.1.202:54336
headscale   | 2024-06-15T03:48:08Z ERR home/runner/work/headscale/headscale/hscontrol/app.go:414 > failed to validate token error="failed to parse ApiKey" client_address=10.0.1.202:54352
headplane   | HeadscaleError: Unauthorized
headplane   |     at pull (file:///app/build/server/index.js?t=1716228566000:846:11)
headplane   |     at processTicksAndRejections (node:internal/process/task_queues:95:5)
headplane   |     at async Promise.all (index 0)
headplane   |     at loader$8 (file:///app/build/server/index.js?t=1716228566000:1927:31)
headplane   |     at Object.callRouteLoader (/app/node_modules/.pnpm/@[email protected][email protected]/node_modules/@remix-run/server-runtime/dist/data.js:66:16)
headplane   |     at /app/node_modules/.pnpm/@[email protected]/node_modules/@remix-run/router/router.ts:4266:21
headplane   |     at callLoaderOrAction (/app/node_modules/.pnpm/@[email protected]/node_modules/@remix-run/router/router.ts:4328:16)
headplane   |     at async Promise.all (index 2)
headplane   |     at callDataStrategyImpl (/app/node_modules/.pnpm/@[email protected]/node_modules/@remix-run/router/router.ts:4169:17)
headplane   |     at callDataStrategy (/app/node_modules/.pnpm/@[email protected]/node_modules/@remix-run/router/router.ts:3503:19) {
headplane   |   status: 500
headplane   | }

[tale]

Are you sure either of you are running Headscale 0.23 beta?

[mitchellkellett]

I'm using 0.23.0-alpha12. I've included my dev docker compose file below.

services:
  db:
    image: postgres:latest
    environment:
      POSTGRES_USER: headscale
      POSTGRES_PASSWORD: password
      POSTGRES_DB: headscale
    ports:
      - 5432:5432
    volumes:
      - ./postgres/data:/var/lib/postgresql/data
  server:
    image: headscale/headscale:v0.23.0-alpha12
    ports:
      - 8080:8080
      - 9090:9090
      - 50443:50443
    volumes:
      - ./prod_headscale:/etc/headscale
    command: serve
    depends_on:
      - db
  ui:
    image: ghcr.io/tale/headplane:0.1.5
    volumes:
      - './headscale:/etc/headscale'
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
    ports:
      - '7070:7070'
    environment:
      # This is always required for Headplane to work
      COOKIE_SECRET: 'Ya1jBVCDeDgjxWg2G92E68HWZ8FyJYqU'
      HEADSCALE_INTEGRATION: 'docker'
      HEADSCALE_CONTAINER: 'headscale-server-1'
      DISABLE_API_KEY_LOGIN: 'false'
      HOST: '0.0.0.0'
      PORT: '7070'
      ACL_FILE: /etc/headscale/acl.yaml
        
      # Overrides the configuration file values if they are set in config.yaml
      # If you want to share the same OIDC configuration you do not need this
      # OIDC_CLIENT_ID: 'headscale'
      # OIDC_ISSUER: 'https://sso.example.com'
      # OIDC_CLIENT_SECRET: 'super_secret_client_secret'

      # This NEEDS to be set with OIDC, regardless of what's in the config
      # This needs to be a very long-lived (999 day) API key used to create
      # shorter ones for OIDC and allow the OIDC functionality to work
      ROOT_API_KEY: '3BqeIbI6bQ.CQZsbKfKkZfnGuIOmI10DJWfywQLjq-Gr_N1tnCZldI'
    depends_on:
      - db
      - server

[mitchellkellett]

Strangely enough, I bit the bullet and deployed it in my "prod" version and it worked seamlessly.

[larrydewey]

I am running Alpha 12, as well. Let me try the Beta release.

[tale]

@larrydewey the issue seems to be related to port numbers. Deploying on a domain might fix it.

[bigbozza]

I had this is exact issue and I can still reproduce this issue in my current setup.

What I tried that didn't work: native linux headscale + native linux headplane running on :3000/admin (non ssl) traefik proxy in front of docker headcale + docker headplane running on :3000/admin (but not proxied by traefik)

What works: traefik proxy in front of docker headcale AND traefik proxied docker headplane...

The major difference in this setup and where I think it needs to be looked into is when using a proxy, I used the endpoint of /admin which is pointing to container port of :3000

But the container itself won't respond on :3000 alone, it wants /admin/ in front of it

this is the config with the traefik labels for clarity:

services:
  headscale:
    image: 'headscale/headscale:0.23.0-alpha10'
    container_name: 'headscale'
    restart: 'unless-stopped'
    command: 'serve'
    volumes:
      - '/etc/headscale:/etc/headscale'
      - '/var/lib/headscale:/var/lib/headscale'
    ports:
      - '8080:8080'
    networks:
      - exposed
    environment:
      TZ: 'Australia/Sydney'
    labels:
      - "traefik.enable=true"
      ## HTTP Routers
      - "traefik.http.routers.headscale.entrypoints=websecure"
      - "traefik.http.routers.headscale.rule=Host(`headscale.domain.com.au`)"
      - "traefik.http.routers.headscalenottls.entrypoints=web"
      - "traefik.http.routers.headscalenottls.rule=Host(`headscale.domain.com.au`)"

      - "traefik.http.routers.headscale.tls=true"
      - "traefik.http.routers.headscale.tls.certresolver=leresolver"
      ## HTTP Services
      - "traefik.http.services.headscale.loadbalancer.server.port=8080"
  headplane:
    container_name: headplane
    image: ghcr.io/tale/headplane:latest
    restart: unless-stopped
    networks:
      - exposed
    ports:
      - '3000:3000'
    volumes:
      - "/etc/headscale/config.yaml:/etc/headscale/config.yaml:rw"
      - "/etc/headscale/acls.json:/etc/headscale/acls.json:rw"
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
     # - "/proc:/proc:rw"
    environment:
      HEADSCALE_URL: 'http://headscale:8080'
      COOKIE_SECRET: 'omitted'
      CONFIG_FILE: '/etc/headscale/config.yaml'
      HEADSCALE_INTEGRATION: 'docker'
      HEADSCALE_CONTAINER: 'headscale'
      API_KEY: 'omitted'
      ACL_FILE: ''
      HOST: '0.0.0.0'
      PORT: '3000'
    labels:
      - "traefik.enable=true"
      ## HTTP Routers
      - "traefik.http.routers.headscale-ui.entrypoints=web,websecure"
      - "traefik.http.routers.headscale-ui.rule=Host(`headscale.domain.com.au`) && PathPrefix(`/admin`)"
      - "traefik.http.routers.headscale-ui.tls.certresolver=leresolver"
      ## HTTP Services
      - "traefik.http.services.headscale-ui.loadbalancer.server.port=3000"
      - "traefik.http.services.headscale-ui.loadbalancer.server.scheme=http"
networks:
  exposed:
    name: exposed

[Chihsiao]

I captured packets with Wireshark and found that the packets sent from headplane to headscale had an incorrect header: Authorization: Bearer undefined.

Upon checking the login response to set cookie, I noticed the Secure flag was present in the Set-Cookie header. This flag is not available for HTTP, which prevents the cookie from being set. As a result, headplane received an undefined key from an empty session and sent it to headscale.

The corresponsding code is in https://github.com/tale/headplane/blob/b8999161a2735fa13e268953a9399aed9b712ecb/app/utils/sessions.ts#L32

To solve it, we can either change the value of secure to false, or make it configurable via an environment variable.