netbird icon indicating copy to clipboard operation
netbird copied to clipboard
verified publisher icon netbirdio

Clients fail to connect to self-hosted NetBird from Cloudflare tunnel over Nginx Proxy Manager

Open lrnd1 opened this issue 1 month ago • 2 comments

Describe the problem

I am running NetBird in docker, using Nginx Proxy Manager, and Authentik.
After creating a Cloudflare tunnel and routing it to NPM, the dashboard loads correctly, but clients cannot connect. When attempting the same from my local network and netbird. DNS record is forwarded directly to NPM, everything works correctly.

To Reproduce

Steps to reproduce the behavior:

  1. run self-hosted version of NetBird
  2. set up Authentik
  3. set up Nginx Proxy Manager
  4. create/configure cloudflare tunnel to route to NPM
  5. accessing dashboard works, connecting clients fail

Expected behavior

Clients should connect the same way as they do from my LAN

Are you using NetBird Cloud?

no

NetBird version

latest version tag - NetBird v0.60.0

Is any other VPN software installed?

no

Debug output

To help us resolve the problem, please attach the following anonymized status output

netbird status -dA

Daemon status: LoginFailed

Run UP command to log in with SSO (interactive login):

 netbird up

If you are running a self-hosted version and no SSO provider has been configured in your Management Server,
you can use a setup-key:

 netbird up --management-url <YOUR_MANAGEMENT_URL> --setup-key <YOUR_SETUP_KEY>

More info: https://docs.netbird.io/how-to/register-machines-using-setup-keys

Create and upload a debug bundle, and share the returned file key:

netbird debug for 1m -AS -U Error: failed to up: up already in progress: current status LoginFailed

Alternatively, create the file only and attach it here manually:

netbird debug for 1m -AS Error: failed to up: up already in progress: current status LoginFailed

Screenshots

Image

Additional context

When attempting to connect, I can see the connection reaching NPM logs and looping: [30/Nov/2025:22:20:42 +0000] - - 301 - POST http netbird.<domain> "/management.ManagementService/GetServerKey" [Client 10.99.22.3] [Length 166] [Gzip -] [Sent-to netbird-dashboard] "grpc-go/1.73.0" "-"

While attempting to connect from my LAN:

[30/Nov/2025:22:29:46 +0000] - 200 200 - POST https netbird.<domain> "/management.ManagementService/GetServerKey" [Client 192.168.148.1] [Length 61] [Gzip -] [Sent-to netbird-dashboard] "grpc-go/1.73.0" "-"
[30/Nov/2025:22:29:46 +0000] - 200 200 - POST https netbird.<domain> "/management.ManagementService/Login" [Client 192.168.148.1] [Length 403] [Gzip -] [Sent-to netbird-dashboard] "grpc-go/1.73.0" "-"
[30/Nov/2025:22:29:46 +0000] - 200 200 - POST https netbird.<domain> "/management.ManagementService/GetServerKey" [Client 192.168.148.1] [Length 61] [Gzip -] [Sent-to netbird-dashboard] "grpc-go/1.73.0" "-"
[30/Nov/2025:22:29:47 +0000] - 200 200 - POST https netbird.<domain> "/management.ManagementService/Login" [Client 192.168.148.1] [Length 403] [Gzip -] [Sent-to netbird-dashboard] "grpc-go/1.73.0" "-"
[30/Nov/2025:22:29:47 +0000] - - 301 - GET https netbird.<domain> "/relay" [Client 192.168.148.1] [Length 166] [Gzip -] [Sent-to netbird-dashboard] "Go-http-client/1.1" "-"
[30/Nov/2025:22:29:47 +0000] - 404 404 - GET https netbird.<domain> "/relay/" [Client 192.168.148.1] [Length 19] [Gzip -] [Sent-to netbird-dashboard] "Go-http-client/1.1" "https://netbird.<domain>/relay"
[30/Nov/2025:22:29:47 +0000] - - 301 - GET https netbird.<domain> "/relay" [Client 192.168.148.1] [Length 166] [Gzip -] [Sent-to netbird-dashboard] "Go-http-client/1.1" "-"
[30/Nov/2025:22:29:47 +0000] - 404 404 - GET https netbird.<domain> "/relay/" [Client 192.168.148.1] [Length 19] [Gzip -] [Sent-to netbird-dashboard] "Go-http-client/1.1" "https://netbird.<domain>/relay"
[30/Nov/2025:22:29:47 +0000] - 200 200 - POST https netbird.<domain> "/management.ManagementService/GetServerKey" [Client 192.168.148.1] [Length 61] [Gzip -] [Sent-to netbird-dashboard] "grpc-go/1.73.0" "-"
[30/Nov/2025:22:29:48 +0000] - 200 200 - POST https netbird.<domain> "/signalexchange.SignalExchange/Send" [Client 192.168.148.1] [Length 5] [Gzip -] [Sent-to netbird-dashboard] "grpc-go/1.73.0" "-"
[30/Nov/2025:22:29:49 +0000] - - 301 - GET https netbird.<domain> "/relay" [Client 192.168.148.1] [Length 166] [Gzip -] [Sent-to netbird-dashboard] "Go-http-client/1.1" "-"
[30/Nov/2025:22:29:49 +0000] - 404 404 - GET https netbird.<domain> "/relay/" [Client 192.168.148.1] [Length 19] [Gzip -] [Sent-to netbird-dashboard] "Go-http-client/1.1" "https://netbird.<domain>/relay"
[30/Nov/2025:22:29:51 +0000] - 200 200 - POST https netbird.<domain> "/signalexchange.SignalExchange/Send" [Client 192.168.148.1] [Length 5] [Gzip -] [Sent-to netbird-dashboard] "grpc-go/1.73.0" "-"
[30/Nov/2025:22:29:53 +0000] - - 301 - GET https netbird.<domain> "/relay" [Client 192.168.148.1] [Length 166] [Gzip -] [Sent-to netbird-dashboard] "Go-http-client/1.1" "-"
[30/Nov/2025:22:29:53 +0000] - 404 404 - GET https netbird.<domain> "/relay/" [Client 192.168.148.1] [Length 19] [Gzip -] [Sent-to netbird-dashboard] "Go-http-client/1.1" "https://netbird.<domain>/relay"


- Cloudflare tunnel connects to NPM via HTTPS on port 443. 
- gRPC in cloudflare is enabled
- Origin Server Name: netbird.<domain>
- No TLS Verify: enabled
- HTTP2 connection: enabled
- HTTP Host Header: netbird.<domain>
- Disable Chunked Encoding: enabled
- Connect Timeout: 120 s
- Idle Connection Expiration Time: 300 s

Everything runs on the same host, which is a mac mini, running Orbstack. I have tried many combination of settings while attempting to fix things, so my config might seem slightly messy.

NPM config:

netbird. -> netbird-dashboard:80 Settings:

proxy_set_header        X-Real-IP $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header        X-Scheme $scheme;
proxy_set_header        X-Forwarded-Proto https;
proxy_set_header        X-Forwarded-Host $host;
grpc_set_header         X-Forwarded-For $proxy_add_x_forwarded_for;
underscores_in_headers on;

Custom Locations:

/api -> netbird-management:443 Advanced:

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Host $host;
grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

~ ^/management.ManagementService/* -> netbird-management:443 Advanced:

grpc_pass grpc://netbird-management:443;
grpc_set_header x-forwarded-for $proxy_add_x_forwarded_for;
grpc_set_header host $host;
grpc_read_timeout 3600s;
grpc_send_timeout 3600s;
grpc_socket_keepalive on;

location ~ ^/management\.ManagementService/ {
    grpc_pass grpc://netbird-management:443;
    grpc_set_header x-forwarded-for $proxy_add_x_forwarded_for;
    grpc_set_header host $host;
    grpc_read_timeout 3600s;
    grpc_send_timeout 3600s;
    grpc_socket_keepalive on;
}

~ ^/signalexchange.SignalExchange/ -> netbird-signal:80 Advanced:

grpc_pass grpc://netbird-signal:80;
grpc_set_header x-forwarded-for $proxy_add_x_forwarded_for;
grpc_set_header host $host;
grpc_read_timeout 3600s;
grpc_send_timeout 3600s;
grpc_socket_keepalive on;

location ~ ^/signalexchange\.SignalExchange/ {
    grpc_pass grpc://netbird-signal:80;
    grpc_set_header x-forwarded-for $proxy_add_x_forwarded_for;
    grpc_set_header host $host;
    grpc_read_timeout 3600s;
    grpc_send_timeout 3600s;
    grpc_socket_keepalive on;
}

/relay/ -> netbird-relay:33080 /ws-proxy/signal -> netbird-signal:80 /ws-proxy/management -> netbird-management:443

docker-compose file:

---
services:
  dashboard:
    container_name: netbird-dashboard
    environment:

      - AUTH_AUDIENCE=${NB_ClientID}  # Client ID
      - AUTH_AUTHORITY=${NB_Authority} 
      - AUTH_CLIENT_ID=${NB_ClientID}
      - AUTH_REDIRECT_URI= #/auth
      - AUTH_SILENT_REDIRECT_URI= #/silent-auth
      - AUTH_SUPPORTED_SCOPES=openid profile email offline_access api
      
      # Endpoints
      - NETBIRD_MGMT_API_ENDPOINT=${NB_MGMT_API} 
      - NETBIRD_MGMT_GRPC_API_ENDPOINT=${NB_MGMT_API}
      
      - NETBIRD_TOKEN_SOURCE=accessToken #idToken 
      - NGINX_SSL_PORT=443 
      - USE_AUTH0=false
      - NETBIRD_AUTH_PKCE_DISABLE_PROMPT_LOGIN=true  # fixes #3654 double-prompt
      
      # LetsEncrypt Disable
      - LETSENCRYPT_DOMAIN=
      - LETSENCRYPT_EMAIL=

    #ports:
    #  - "8029:80"
    image: netbirdio/dashboard:latest
    logging:
      driver: "json-file"
      options:
        max-file: "2"
        max-size: "500m"
    networks: 
      - netbird-tunnel
    restart: unless-stopped
    volumes:
      - /Applications/docker/netbird/netbird-letsencrypt:/etc/letsencrypt/
  # Management
  management:
    container_name: netbird-management
    command: [
      "--disable-anonymous-metrics=true",
      "--dns-domain=netbird.<domain>",
      "--log-file", "console",
      "--log-level", "info",
      "--port", "443",
      "--single-account-mode-domain=netbird.<domain>"
      ]
    depends_on:
      dashboard:
        condition: service_started
    environment:
      - NETBIRD_AUTH_DEVICE_AUTH_CLIENT_ID=${NB_ClientID}
      - NETBIRD_AUTH_DEVICE_AUTH_AUDIENCE=${NB_ClientID}
      - NETBIRD_USE_AUTH0=false
      - NETBIRD_DISABLE_LETSENCRYPT=true
    #ports:
    #  - 33073:33073
    #  - 33074:443 #API port
    image: netbirdio/management:latest
    logging:
      driver: "json-file"
      options:
        max-file: "2"
        max-size: "500m"
    networks: 
      - netbird-tunnel
    restart: unless-stopped
    volumes:
      - /Applications/docker/netbird/management.json:/etc/netbird/management.json
      - /Applications/docker/netbird/mgmt:/var/lib/netbird
      - /Applications/docker/netbird/netbird-letsencrypt:/etc/letsencrypt:ro

  # Signal
  signal:
    container_name: netbird-signal
    image: netbirdio/signal:latest
    logging:
      driver: "json-file"
      options:
        max-file: "2"
        max-size: "500m"
    networks: 
      - netbird-tunnel
    #ports:
    #  - "10000:80"
    restart: unless-stopped
    volumes:
      - /Applications/docker/netbird/signal:/var/lib/netbird
      - /Applications/docker/netbird/netbird-letsencrypt:/etc/letsencrypt:ro

  # Relay
  relay:
    container_name: netbird-relay
    environment:
    - NB_AUTH_SECRET=${NB_Relay_Sec}
    - NB_EXPOSED_ADDRESS=${NB_Relay_Addr}
    - NB_LISTEN_ADDRESS=:33080
    - NB_LOG_LEVEL=info
    image: netbirdio/relay:latest
    logging:
      driver: "json-file"
      options:
        max-file: "2"
        max-size: "500m"
    networks: 
      - netbird-tunnel
    #ports:
    #  - 33080:33080
    restart: unless-stopped
      
  # Coturn
  coturn:
    container_name: netbird-coturn
    image: coturn/coturn:latest
    restart: unless-stopped
    volumes:
      - /Applications/docker/netbird/turnserver.conf:/etc/turnserver.conf:ro
    network_mode: host
    command:
      - -c /etc/turnserver.conf
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"

  tunnel:
    container_name: cloudflared-tunnel-netbird
    image: cloudflare/cloudflared
    restart: unless-stopped
    command: tunnel run
    environment:
      - TUNNEL_TOKEN=${MyCFtoken}
    networks:
      - netbird-cf-tunnel
#    volumes:
#      - /Applications/docker/netbird/CFconfig.yml:/etc/cloudflared/config.yml:ro


networks:
  netbird-tunnel:
    name: netbird-tunnel
  netbird-cf-tunnel:
    name: netbird-cf-tunnel

(NPM is in a different stack, but in the same network with cloudflared-tunnel-netbird)

Relevant env vars:

NB_Endpoint=https://authentik.<domain>/application/o/netbird/.well-known/openid-configuration
NB_Authority=https://authentik.<domain>/application/o/netbird/
NB_MGMT_API=https://netbird.<domain>
NB_Relay_Addr=rels://netbird.<domain>/relay

management.json file:

{
  "Stuns": [
    {
      "Proto": "udp",
      "URI": "stun:netbird-coturn:3478",
      "Username": "",
      "Password": null
    }
  ],
  "TURNConfig": {
    "Turns": [
      {
        "Proto": "udp",
        "URI": "turn:netbird-coturn:3478",
        "Username": "self",
        "Password": "${NB_Relay_Sec}"
      }
    ],
    "CredentialsTTL": "12h",
    "Secret": "${NB_Relay_Sec}",
    "TimeBasedCredentials": false
  },
  "Relay": {
    "Addresses": [
      "rels://netbird.<domain>/relay"
    ],
    "CredentialsTTL": "24h",
    "Secret": "${NB_Relay_Sec}"
  },
  "Signal": {
    "Proto": "https",
    "URI": "netbird.<domain>:443",
    "Username": "",
    "Password": null
  },
  "HttpConfig": {
    "Address": "0.0.0.0:443",
    "AuthIssuer": "https://authentik.<domain>/application/o/netbird/",
    "AuthAudience": "${NB_ClientID}",
    "AuthKeysLocation": "https://authentik.<domain>/application/o/netbird/jwks/",
    "OIDCConfigEndpoint": "https://authentik.<domain>/application/o/netbird/.well-known/openid-configuration"
  },
  "IdpManagerConfig": {
    "ManagerType": "authentik",
    "ClientConfig": {
      "Issuer": "https://authentik.<domain>/application/o/netbird/",
      "TokenEndpoint": "https://authentik.<domain>/application/o/token/",
      "ClientID": "${NB_ClientID}",
      "ClientSecret": "<Authentik service user token>",
      "GrantType": "client_credentials"
    },
    "ExtraConfig": {
      "Username": "NetBird",
      "Password": "<Authentik service user app pass>"
    },
    "Auth0ClientCredentials": null,
    "AzureClientCredentials": null,
    "KeycloakClientCredentials": null,
    "ZitadelClientCredentials": null
  },
  "DeviceAuthorizationFlow": {
    "Provider": "hosted",
    "ProviderConfig": {
      "Audience": "${NB_ClientID}",
      "AuthorizationEndpoint": "https://authentik.<domain>/application/o/netbird/authorize/",
      "Domain": "authentik.<domain>",
      "ClientID": "${NB_ClientID}",
      "ClientSecret": "<Authentik service user token>",
      "TokenEndpoint": "https://authentik.<domain>/application/o/token/",
      "DeviceAuthEndpoint": "https://authentik.<domain>/application/o/device/",
      "Scope": "openid profile email offline_access api",
      "UseIDToken": false,
      "RedirectURLs": [
        "http://localhost:53000"
      ]
    }
  },
  "PKCEAuthorizationFlow": {
      "ProviderConfig": {
            "ClientID": "${NB_ClientID}",
            "ClientSecret": "<Authentik service user token>",
            "Domain": "authentik.<domain>/application/o/netbird/",
            "Audience": "${NB_ClientID}",
            "TokenEndpoint": "https://authentik.<domain>/application/o/netbird/token/",
            "DeviceAuthEndpoint": "",
            "AuthorizationEndpoint": "https://authentik.<domain>/application/o/netbird/authorize/",
            "Scope": "openid profile email offline_access api",
            "UseIDToken": false,
            "RedirectURLs": [
                "http://localhost:53000"
            ]
      }
    },
   "ReverseProxy": {
      "TrustedHTTPProxies": [ 
        "0.0.0.0/0"
      ],
      "TrustedHTTPProxiesCount": 0,
      "TrustedPeers": [
        "0.0.0.0/0"
      ]
    },
  "Datadir": "",
  "DataStoreEncryptionKey": "<random pass>"
}

I spent many hours troubleshooting, and would appreciate any ideas that I could try to fix it. issue #3110 is very similar to my case, yet I couldn't find a solution from reading it.

Have you tried these troubleshooting steps?

  • [ x] Reviewed client troubleshooting (if applicable)
  • [ x] Checked for newer NetBird versions
  • [ x] Searched for similar issues on GitHub (including closed ones)
  • [ x] Restarted the NetBird client
  • [ ] Disabled other VPN software
  • [ ] Checked firewall settings

lrnd1 avatar Nov 30 '25 23:11 lrnd1

I tried setting up something similar using traefik and cloudflare tunnel, I don't think the tunnel supports grpc?.

I was getting similar errors "grpc context ended early, error: context canceled" and "retrying Login to the Management service in 1. 149892488s due to error rpc error code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key"

I ended up setting up a vps with traefik and netbird and it worked pretty much straight away using the reverse proxy ports only.

Selmaks avatar Dec 07 '25 23:12 Selmaks

I tried setting up something similar using traefik and cloudflare tunnel, I don't think the tunnel supports grpc?.

I was getting similar errors "grpc context ended early, error: context canceled" and "retrying Login to the Management service in 1. 149892488s due to error rpc error code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key"

I ended up setting up a vps with traefik and netbird and it worked pretty much straight away using the reverse proxy ports only.

Yeah, I pretty much had a similar experience. After trying 3 different reverse proxies, testing with grpcurl, trailers were always missing when routing through cloudflare. Even when directly routing /management.Managementservice/ from cloudflare tunnel to the container.

Yesterday I ended up splitting the routing into different paths. /* /api/* and /relay remained on cloudflare, and created a separate DNS without cloudflare proxy (grey cloud) that points the rest of the routes directly to my server with router port forwarding.
Now clients can connect, but I cannot use the same DNS for both the dashboard access and netbird clients.

lrnd1 avatar Dec 08 '25 07:12 lrnd1