Headscale behind TLS causes invalid API key issues.

[joedima]

Hi

I am writing this because I have been trying to configure headplane to work with headscale for a few days and I am very frustrated. I am doing something wrong and I do not understand what.

So I have both headscale and headplane running in docker on a VPS server. My reverse proxy is Caddy. I have configured headplane to log in at my.domain.com/admin. The enter API dialog shows up and when I enter the key, it tells me that the key is invalid. Headscale log is telling me that it is missing 'Bearer' prefix from the authorization header.

ERR home/runner/work/headscale/headscale/hscontrol/app.go:357 > missing "Bearer " prefix in "Authorization" header client_address=172.18.0.2:54850

However, when I run a local headscale in a local container and headplane in debug mode with 'pnpm dev', this does not happen. The key is recognized immediately.

If someone is kind enough to point me in the right direction, it would be greatly appreciate it.

Thank you.

[tale]

Can you send your exact configuration please.

[joedima]

This is from my docker_compose.yml:

headplane: container_name: headplane image: ghcr.io/tale/headplane:0.3.9 restart: unless-stopped volumes: - ./data:/var/lib/headscale - ./config:/etc/headscale - /var/run/docker.sock:/var/run/docker.sock:ro ports: - '3000:3000' environment: DEBUG: 'true' COOKIE_SECRET: 'cookie_secret' # HEADSCALE_URL: 'https://my.domain.net' HEADSCALE_INTEGRATION: 'docker' HEADSCALE_CONTAINER: 'headscale' DISABLE_API_KEY_LOGIN: 'true' HOST: '0.0.0.0' PORT: '3000' # CONFIG_FILE: '/etc/headscale/config.yaml' COOKIE_SECURE: 'true' # ROOT_API_KEY: 'zabcdefghijklmnopqrstuvwxyz' ROOT_API_KEY: 'root_api_key' networks: - proxy_network

Please let me know if you need more information. Thank you for your help.

[joedima]

The program requires ROOT_API_KEY no matter what. If it is not specified, it will go in a loop and not serve pages for log in. But since I do not have OIDC set up yet, I do not think it should be need it. It is also confusing because in the documentation is mentioned that it can be optional.

[tale]

That is most likely a bug then, let me investigate and I'll get back to you.

[joedima]

If you want me to help you with testing, I would be glad.

Also, I will try to do the Authelia setup since with the latest updates, they added a new layer to AuthN/AuthZ. I will let you know what I find.

[RJDavison]

Is there any update to this? I'm having the exact same problem using Caddy with virtually the same config.

Requires ROOT_API_KEY even if you set this to false. Also cant get past the API key as it keeps saying that its invalid.

[chan-a]

Same issue here. Here's my docker-compose.yml

  caddy:
    image: caddy:latest
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - proxy_network

  headscale:
    image: headscale/headscale:stable
    volumes:
      - ./headscale/config:/etc/headscale
      - headscale-data:/var/lib/headscale
    expose:
      - "8080"
      - "9090"
    command: serve
    restart: unless-stopped
    networks:
      - proxy_network

  headplane:
    container_name: headplane
    image: ghcr.io/tale/headplane:latest
    restart: unless-stopped
    volumes:
      - ./headscale/config:/etc/headscale
      - headscale-data:/var/lib/headscale
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
    environment:
      # This is always required for Headplane to work
      COOKIE_SECRET: 'asdfsadifasdoifbasdiof'
      HEADSCALE_INTEGRATION: 'docker'
      HEADSCALE_CONTAINER: 'headscale'
      DISABLE_API_KEY_LOGIN: 'true'
      HOST: '0.0.0.0'
      PORT: '3000'
      ROOT_API_KEY: 'abcd'
    networks:
      - proxy_network

And caddy file

mydomain {
    reverse_proxy headscale:8080
    encode gzip
}

Note that I did manage to get it to run with the "basic" configuration, note the docker integration has all been commented out

  headplane:
    container_name: headplane
    image: ghcr.io/tale/headplane:latest
    restart: unless-stopped
    # volumes:
    #   - ./headscale/config:/etc/headscale
    #   - headscale-data:/var/lib/headscale
    #   - '/var/run/docker.sock:/var/run/docker.sock:ro'
    # ports:
    #   - '3000:3000'
    environment:
      HEADSCALE_URL: 'http://headscale:8080'
      COOKIE_SECRET: 'asdfsadifasdoifbasdiof'
      # HEADSCALE_INTEGRATION: 'docker'
      # HEADSCALE_CONTAINER: 'headscale'
      DISABLE_API_KEY_LOGIN: 'true'
      HOST: '0.0.0.0'
      PORT: '3000'
      ROOT_API_KEY: 'abcd'
    networks:
      - proxy_network

[RJDavison]

What worked for me was to add a propert API_KEY despite the config instructions saying that its not needed and having the DISABLE_API_KEY_LOGIN set to true.

ROOT_API_KEY: 'zabcdefghijklmnopqrstuvwxyz'

[GraphicHamster]

I've also been struggling to get headplane to even launch (docker) I can confirm that giving it some gibberish for ROOT_API_KEY got it to launch

[tale]

ROOT_API_KEY needs to be a valid Headplane generated key.

[zmweske]

Does the API key need to be an api key for your oidc provider or a preauth key for your headscale server?

EDIT: It seems to be a headscale apikeys create product

[RJDavison]

The headscale server. Just make sure you create it with a long expiry.

On Mon, 10 Feb 2025, 06:08 zmweske, @.***> wrote:

Does the API key need to be an api key for your oidc provider or a preauth key for your headscale server?

— Reply to this email directly, view it on GitHub https://github.com/tale/headplane/issues/82#issuecomment-2646821142, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMVNQ4DVQQRAX56IBB2QS32PAJ3LAVCNFSM6AAAAABVBUNIN2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNBWHAZDCMJUGI . You are receiving this because you commented.Message ID: @.***>

[Herve-M]

The headscale server. Just make sure you create it with a long expiry. …

According to the doc. ROOT_API_KEY is related to OIDC provider, not headscale,. Maybe it should be updated?

[tale]

Yes the docs are bad, and I will fix them.

[wx2020]

@tale By the way, can you provide an example of a configuration file in the docs when using nginx or caddy as a front proxy? Because I didn't find it anywhere. Thanks a lot!

[Jonybat]

I suppose this issue also applies to the native linux integration. I tried multiple combinations of the env vars mentioned above and config.yaml/.env, with no luck

[JustAnotherRandomUsername]

same issue, can not auth with API key...

have a two docker containers with headscale version 0.25.1 and headplane version 0.5.3

headplane config.yaml:

server:
    host: "0.0.0.0"
    port: 3000
    cookie_secret: "WzxNN1fKzn6eDGzZUdtmkgybPdE1lbv7"
    cookie_secure: false
headscale:
    url: "https://headscale:50443"
    config_path: "/etc/headscale/config.yaml"
    config_strict: true
integration:
    docker:
        enabled: true
        container_name: "headscale"
        socket: "unix:///var/run/docker.sock"
    kubernetes:
        enabled: false
        validate_manifest: true
        pod_name: "headscale"
    proc:
        enabled: false

headscale config.yaml:

server_url: https://lolithebest.ddns.net:8080
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 0.0.0.0:9090
grpc_listen_addr: 0.0.0.0:50443
grpc_allow_insecure: false

noise:
  private_key_path: /var/lib/headscale/noise_private.key

prefixes:
  v4: 100.64.0.0/10
  v6: fd7a:115c:a1e0::/48
  allocation: sequential

derp:
  server:
    enabled: false
    region_id: 999
    region_code: "headscale"
    region_name: "Headscale Embedded DERP"
    stun_listen_addr: "0.0.0.0:3478"
    private_key_path: /var/lib/headscale/derp_server_private.key
    automatically_add_embedded_derp_region: true
    ipv4: 1.2.3.4
    ipv6: 2001:db8::1

  urls:
    - https://controlplane.tailscale.com/derpmap/default

  paths: []
  auto_update_enabled: true
  update_frequency: 1h

disable_check_updates: false
ephemeral_node_inactivity_timeout: 10m

database:
  type: sqlite
  debug: false

  gorm:
    prepare_stmt: true
    parameterized_queries: true
    skip_err_record_not_found: true
    slow_threshold: 1000

  sqlite:
    path: /var/lib/headscale/db.sqlite
    write_ahead_log: true
    wal_autocheckpoint: 1000

acme_url: https://acme-v02.api.letsencrypt.org/directory
acme_email: ""
tls_letsencrypt_hostname: ""
tls_letsencrypt_cache_dir: /var/lib/headscale/cache
tls_letsencrypt_challenge_type: HTTP-01
tls_letsencrypt_listen: ":http"
tls_cert_path: "/var/lib/headscale/ssl/lolithebest.ddns.net.crt"
tls_key_path: "/var/lib/headscale/ssl/lolithebest.ddns.net.key"

log:
  format: text
  level: info

policy:
  mode: file
  path: ""

dns:
  magic_dns: true
  base_domain: mydns.lan

  nameservers:
    global:
      - 1.1.1.1
      - 1.0.0.1
      - 2606:4700:4700::1111
      - 2606:4700:4700::1001
    split:
      {}

  search_domains: []
  extra_records: []

unix_socket: /var/run/headscale/headscale.sock
unix_socket_permission: "0770"

logtail:
  enabled: false

randomize_client_port: false

API key was generated with command: docker compose exec headscale headscale apikeys create

so I can see it with docker compose exec headscale headscale apikeys list

ID | Prefix  | Expiration          | Created
1  | pviT6HD | 2025-06-05 23:22:00 | 2025-03-07 23:22:00

docker-compose.yml:

services:
  headplane:
    image: ghcr.io/tale/headplane:0.5.3
    container_name: headplane
    restart: unless-stopped
    ports:
      - '3000:3000'
    volumes:
      - './headplane-config/config.yaml:/etc/headplane/config.yaml'
      - './headscale-config/config.yaml:/etc/headscale/config.yaml'
      - '/var/run/docker.sock:/var/run/docker.sock:ro'
    environment:
      ROOT_API_KEY: 'pviT6HD.x9P7E7Wg7LsodqrV97fsa1If_CQZu6B0'
      COOKIE_SECRET: 'WzxNN1fKzn6eDGzZUdtmkgybPdE1lbv7'
      DISABLE_API_KEY_LOGIN: 'true'

  headscale:
    image: headscale/headscale:0.25.1
    container_name: headscale
    restart: unless-stopped
    command: serve
    ports:
      - '8080:8080'
    volumes:
      - './headscale-data:/var/lib/headscale'
      - './headscale-config:/etc/headscale'

Using DISABLE_API_KEY_LOGIN: 'false' also not allow me to login

trying curl -sS -k -v -u "pviT6HD.x9P7E7Wg7LsodqrV97fsa1If_CQZu6B0:" https://headscale:50443

got response: curl: (1) Received HTTP/0.9 when not allowed

[JustAnotherRandomUsername]

ok, my bad

the gRPC (CLI) is not an API

works well with headplane config.yaml:

headscale:
    url: "http://headscale:8080"

headscale config.yaml:

server_url: http://lolithebest.ddns.net:8080
...
tls_cert_path: ""
tls_key_path: ""

and removed environment from docker-compose.yml

so problem appears when using certificates and https connection

[ulviallahverdi]

@JustAnotherRandomUsername , thanks forcsharing then does it work for you now in this case ? You just disabled tls then it started to work?

[JustAnotherRandomUsername]

@JustAnotherRandomUsername , thanks forcsharing then does it work for you now in this case ? You just disabled tls then it started to work?

yes, seems the TLS issue

due to experiments with certificates I've catch those strings in logs:

headplane  |   [cause]: Error: unable to verify the first certificate
headplane  |       at TLSSocket.onConnectSecure (node:_tls_wrap:1679:34)
headplane  |       at TLSSocket.emit (node:events:518:28)
headplane  |       at TLSSocket._finishInit (node:_tls_wrap:1078:8)
headplane  |       at ssl.onhandshakedone (node:_tls_wrap:864:12) {
headplane  |     code: 'UNABLE_TO_VERIFY_LEAF_SIGNATURE'
headplane  |   }

looks like container with headplane cannot verify headscale certificate

I used acme.sh to create this one

[ulviallahverdi]

Thanks for explanation, when I turn of TLS all fine works, but did you able to use HEADSCALE with TLS through HEADPLANE? because you mentioned acme.sh.

[JustAnotherRandomUsername]

Thanks for explanation, when I turn of TLS all fine works, but did you able to use HEADSCALE with TLS through HEADPLANE? because you mentioned acme.sh.

yes, eventually I've fix it

[ulviallahverdi]

Thanks for explanation, when I turn of TLS all fine works, but did you able to use HEADSCALE with TLS through HEADPLANE? because you mentioned acme.sh.

yes, eventually I've fix it

How did you fix it? Did it work with tls ?

[JustAnotherRandomUsername]

Thanks for explanation, when I turn of TLS all fine works, but did you able to use HEADSCALE with TLS through HEADPLANE? because you mentioned acme.sh.

yes, eventually I've fix it

How did you fix it? Did it work with tls ?

I've done this creating concatenated certificate (cert + intermediate + ca) Yes, it work with TLS now

[ulviallahverdi]

Did you this for tailscale server or for headplane? Sorry for just i stuck, I did auto ssl by headscale config then i installed headplane on it. Thats why i am asking.

[JustAnotherRandomUsername]

Did you this for tailscale server or for headplane? Sorry for just i stuck, I did auto ssl by headscale config then i installed headplane on it. Thats why i am asking.

okay, just put together all my comments:

1. topic named: Cannot log in using API Key...
2. my comment: same issue, can not auth with API key...
3. my comment: ok, my bad the gRPC (CLI) is not an API you asked about disabled tls
4. my comment for you: yes, seems the TLS issue you asked about: did you able to use HEADSCALE with TLS through HEADPLANE
5. my comment for you: yes, eventually I've fix it you asked about: How did you fix it? Did it work with tls ?
6. my comment for you: I've done this creating concatenated certificate (cert + intermediate + ca) Yes, it work with TLS now

please read my comment marked number 4 : looks like container with headplane cannot verify headscale certificate

I'm not sure is this similar issue to you issue or not

[tale]

@JustAnotherRandomUsername based on your findings, would you feel comfortable updating the docs in Configuration.md with guidance on running Headscale under TLS, since it appears to be the root problem? I'd also like to add some runtime code checking for this, but I'm not sure what exactly the correct behavior needs to be.

[JustAnotherRandomUsername]

@JustAnotherRandomUsername based on your findings, would you feel comfortable updating the docs in Configuration.md with guidance on running Headscale under TLS, since it appears to be the root problem? I'd also like to add some runtime code checking for this, but I'm not sure what exactly the correct behavior needs to be.

1. I'm not sure about my attempt to follow right format of Configuration.md
2. May be better take this manual separately from Configuration.md
3. English is not my native language (sorry for mistakes)
4. Do all what you want: https://pastebin.com/KmXkV89p

[tale]

Ok thank you for the pastebin, I'll get to it when I can it's very helpful!

[ulviallahverdi]

@JustAnotherRandomUsername based on your findings, would you feel comfortable updating the docs in Configuration.md with guidance on running Headscale under TLS, since it appears to be the root problem? I'd also like to add some runtime code checking for this, but I'm not sure what exactly the correct behavior needs to be.

1. I'm not sure about my attempt to follow right format of Configuration.md
2. May be better take this manual separately from Configuration.md
3. English is not my native language (sorry for mistakes)
4. Do all what you want: https://pastebin.com/KmXkV89p

I am able to use now but problem is when I make cookie_secure true i can not login to headplane when I make it false it works. how can I fix it as well?