netbird icon indicating copy to clipboard operation
netbird copied to clipboard

Published 20 hours ago • verified publisher icon netbirdio

all clients lose connection to management server

Open scroguard opened this issue 5 months ago • 8 comments

as of yesterday afternoon, all of my clients have lost connectivity to the management server. all of them show the following error when attempting to connect:

2025-06-07T22:08:24Z WARN client/cmd/root.go:257: retrying Login to the Management service in 1.090533137s due to error rpc error: code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key 2025-06-07T22:08:25Z WARN client/cmd/root.go:257: retrying Login to the Management service in 1.56593389s due to error rpc error: code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key 2025-06-07T22:08:27Z WARN client/cmd/root.go:257: retrying Login to the Management service in 1.68838013s due to error rpc error: code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key 2025-06-07T22:08:29Z WARN client/cmd/root.go:257: retrying Login to the Management service in 1.777346067s due to error rpc error: code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key 2025-06-07T22:08:30Z WARN client/cmd/root.go:257: retrying Login to the Management service in 4.633737091s due to error rpc error: code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key 2025-06-07T22:08:35Z WARN client/cmd/root.go:257: retrying Login to the Management service in 10.758309078s due to error rpc error: code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key 2025-06-07T22:08:46Z WARN client/cmd/root.go:257: retrying Login to the Management service in 7.401033378s due to error rpc error: code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key Error: login backoff cycle failed: rpc error: code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key

this is a self-hosted install that used the 5-minute quickstart install guide. if i roll the management server vm to a backup i have, everyone comes back online for about an hour and then it all dies again.

To Reproduce

Steps to reproduce the behavior:

  1. unknown - cannot currently determine a cause for the issue.

Expected behavior clients should stay connected.

Are you using NetBird Cloud? - no. self-hosted that was setup using quickcstart guide

NetBird version server is on latest release, most clients are also on latest release. otherwise it's a mixture of 0.43.2, 0.38.0, 0.39.2 and 0.37.1.

Is any other VPN software installed?

no

Debug output

To help us resolve the problem, please attach the following anonymized status output

netbird status -dA - this doesn't work as a netbird up will not connect.

Create and upload a debug bundle, and share the returned file key:

netbird debug for 1m -AS -U - can't as current status is LoginFailed

Alternatively, create the file only and attach it here manually:

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Have you tried these troubleshooting steps?

no firewall changes. tried restarting netbird management services. tried rebooting netbird management vm. tried rolling back to a known working backup, clients only stayed online for about an hour and then died again.

scroguard avatar Jun 07 '25 22:06 scroguard

i’ve had a chance to dig a little bit, and when i follow the docker compose logs, caddy repeats this over and over again for each client attempting to connect:

caddy-1 | {"level":"debug","ts":1749362859.288349,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"management:80","duration":0.000987389,"request":{"remote_ip":"50.193.208.183","remote_port":"36286","client_ip":"50.193.208.183","proto":"HTTP/2.0","method":"POST","host":"farnsworth.justagenericdomain.com:443","uri":"/management.ManagementService/GetServerKey","headers":{"X-Forwarded-Host":["farnsworth.justagenericdomain.com:443"],"Grpc-Timeout":["4999979u"],"Content-Type":["application/grpc"],"User-Agent":["grpc-go/1.64.1"],"Te":["trailers"],"X-Forwarded-For":["50.193.208.183"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"farnsworth.justagenericdomain.com"}},"headers":{"Content-Length":["19"],"Date":["Sun, 08 Jun 2025 06:07:39 GMT"],"Content-Type":["text/plain; charset=utf-8"],"X-Content-Type-Options":["nosniff"]},"status":404} caddy-1 | {"level":"debug","ts":1749362859.3012595,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"signal:10000","total_upstreams":1}

scroguard avatar Jun 08 '25 06:06 scroguard

i was able to get a client to attempt to connect and snag a log snippet from it for a little more info.

Image

scroguard avatar Jun 08 '25 06:06 scroguard

Can you share the management logs?

mlsmaycon avatar Jun 08 '25 17:06 mlsmaycon

management-1 | 2025-06-08T15:44:44Z INFO [context: SYSTEM] management/cmd/management.go:510: loading OIDC configuration from the provided IDP configuration endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration management-1 | Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration Get "https://farnsworth.justagenericdomain.com/.well-known/openid-configuration": EOF management-1 | 2025-06-08T15:44:45Z INFO [context: SYSTEM] management/cmd/management.go:510: loading OIDC configuration from the provided IDP configuration endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | 2025-06-08T15:44:45Z INFO [context: SYSTEM] management/cmd/management.go:510: loading OIDC configuration from the provided IDP configuration endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | 2025-06-08T15:44:46Z INFO [context: SYSTEM] management/cmd/management.go:510: loading OIDC configuration from the provided IDP configuration endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | 2025-06-08T15:44:47Z INFO [context: SYSTEM] management/cmd/management.go:510: loading OIDC configuration from the provided IDP configuration endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | 2025-06-08T15:44:50Z INFO [context: SYSTEM] management/cmd/management.go:510: loading OIDC configuration from the provided IDP configuration endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | 2025-06-08T15:44:53Z INFO [context: SYSTEM] management/cmd/management.go:510: loading OIDC configuration from the provided IDP configuration endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/cmd/management.go:510: loading OIDC configuration from the provided IDP configuration endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/cmd/management.go:515: loaded OIDC configuration from the provided IDP configuration endpoint: https://farnsworth.justagenericdomain.com/.well-known/openid-configuration management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/cmd/management.go:517: overriding HttpConfig.AuthIssuer with a new value https://farnsworth.justagenericdomain.com, previously configured value: https://farnsworth.justagenericdomain.com management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/cmd/management.go:521: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://farnsworth.justagenericdomain.com/oauth/v2/keys, previously configured value: https://farnsworth.justagenericdomain.com/oauth/v2/keys management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/cmd/management.go:526: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://farnsworth.justagenericdomain.com/oauth/v2/token, previously configured value: https://farnsworth.justagenericdomain.com/oauth/v2/token management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/cmd/management.go:529: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://farnsworth.justagenericdomain.com/oauth/v2/device_authorization, previously configured value: https://farnsworth.justagenericdomain.com/oauth/v2/device_authorization management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/cmd/management.go:537: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: farnsworth.justagenericdomain.com, previously configured value: farnsworth.justagenericdomain.com management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/cmd/management.go:547: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://farnsworth.justagenericdomain.com/oauth/v2/token, previously configured value: https://farnsworth.justagenericdomain.com/oauth/v2/token management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/cmd/management.go:550: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://farnsworth.justagenericdomain.com/oauth/v2/authorize, previously configured value: https://farnsworth.justagenericdomain.com/oauth/v2/authorize management-1 | 2025-06-08T15:45:00Z INFO management/cmd/management.go:557: Relay addresses: [rels://farnsworth.justagenericdomain.com:443] management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/telemetry/app_metrics.go:193: enabled application metrics and exposing on http://0.0.0.0:9090 management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/store/store.go:256: using SQLite store engine management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/store/sql_store.go:89: Set max open db connections to 1 management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/migration/migration.go:257: No plain setup keys found in table setup_keys, no migration needed management-1 | 2025-06-08T15:45:00Z INFO management/server/migration/migration.go:295: Migration of plain setup key to hashed setup key completed management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty enabled found in table network_resources, no migration needed management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty enabled to default value in table network_resources completed management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty enabled found in table network_routers, no migration needed management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty enabled to default value in table network_routers completed management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/activity/store/sql_store.go:260: using sqlite as activity event store engine management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty name found in table deleted_users, no migration needed management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty name to default value in table deleted_users completed management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty enc_algo found in table deleted_users, no migration needed management-1 | 2025-06-08T15:45:00Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty enc_algo to default value in table deleted_users completed management-1 | 2025-06-08T15:45:01Z INFO [context: SYSTEM] management/cmd/management.go:197: geolocation service has been initialized from /var/lib/netbird/ management-1 | 2025-06-08T15:45:01Z INFO [context: SYSTEM] management/server/account_request_buffer.go:45: set account request buffer interval to 100ms management-1 | 2025-06-08T15:45:01Z WARN [context: SYSTEM] management/server/account.go:249: failed to parse peer update interval, using default value 1ms: strconv.Atoi: parsing "": invalid syntax management-1 | 2025-06-08T15:45:01Z INFO [context: SYSTEM] management/server/account.go:265: set peer update buffer interval to 1ms management-1 | 2025-06-08T15:45:01Z INFO [context: SYSTEM] management/server/account.go:213: single account mode enabled, accounts number 1 management-1 | 2025-06-08T15:45:01Z WARN management/server/token_mgr.go:61: TURN credentials TTL is not set or invalid, using default value 12h0m0s management-1 | 2025-06-08T15:45:01Z WARN [context: SYSTEM] management/cmd/management.go:222: TrustedPeers are configured to default value '0.0.0.0/0', '::/0'. This allows connection IP spoofing. management-1 | 2025-06-08T15:45:01Z INFO [context: SYSTEM] management/cmd/management.go:318: running gRPC backward compatibility server: [::]:33073 management-1 | 2025-06-08T15:45:01Z INFO [context: SYSTEM] management/cmd/management.go:350: management server version 0.46.0 management-1 | 2025-06-08T15:45:01Z INFO [context: SYSTEM] management/cmd/management.go:351: running HTTP server and gRPC server on the same port: [::]:80 management-1 | 2025-06-08T15:45:02Z INFO [context: SYSTEM] management/server/account.go:544: 1 entries received from IdP management management-1 | 2025-06-08T15:45:02Z INFO [context: SYSTEM] management/server/account.go:575: warmed up IDP cache with 1 entries for 1 accounts

scroguard avatar Jun 08 '25 21:06 scroguard

@scroguard there was a brief period with request to your idp failing:

management-1 | Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration Get "https://farnsworth.justagenericdomain.com/.well-known/openid-configuration": EOF
management-1 | 2025-06-08T15:44:45Z INFO [context: SYSTEM] management/cmd/management.go:510: loading OIDC configuration from the provided IDP configuration endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration
management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response:

do you have other logs after the last line you sent? you can try the following:

docker compose logs management 2>&1 | grep -i error

mlsmaycon avatar Jun 08 '25 21:06 mlsmaycon

let me clarify - all clients are still unable to connect. here is the result of the log/grep command:

management-1 | Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration Get "https://farnsworth.justagenericdomain.com/.well-known/openid-configuration": EOF management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response:

strange thing is - when i load the url in a browser, it responds correctly.

scroguard avatar Jun 09 '25 00:06 scroguard

The grep command I sent you before has an extra r, please run this:

docker compose logs management 2>&1 | egrep -i 'erro|warn'

As for the OIDC configuration error, can you run the following commands from the management service container?

apt update && apt install -y curl
curl -v https://farnsworth.justagenericdomain.com/.well-known/openid-configuration

You can access the container with:

docker compose exec -ti management

If the curl command fails, try running on the docker host too. If it fails on both, it is possible that it is going through or to another node?

mlsmaycon avatar Jun 09 '25 00:06 mlsmaycon

the curl command worked from within the container without any issue:

root@1c09d994955e:/# curl -v https://farnsworth.justagenericdomain.com/.well-known/openid-configuration

  • Host farnsworth.justagenericdomain.com:443 was resolved.
  • IPv6: (none)
  • IPv4: 50.193.208.182
  • Trying 50.193.208.182:443...
  • Connected to farnsworth.justagenericdomain.com (50.193.208.182) port 443
  • ALPN: curl offers h2,http/1.1
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
  • ALPN: server accepted h2
  • Server certificate:
  • subject: CN=farnsworth.justagenericdomain.com
  • start date: Jun 6 23:52:28 2025 GMT
  • expire date: Sep 4 23:52:27 2025 GMT
  • subjectAltName: host "farnsworth.justagenericdomain.com" matched cert's "farnsworth.justagenericdomain.com"
  • issuer: C=US; O=Let's Encrypt; CN=E5
  • SSL certificate verify ok.
  • Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
  • Certificate level 1: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using sha256WithRSAEncryption
  • Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • using HTTP/2
  • [HTTP/2] [1] OPENED stream for https://farnsworth.justagenericdomain.com/.well-known/openid-configuration
  • [HTTP/2] [1] [:method: GET]
  • [HTTP/2] [1] [:scheme: https]
  • [HTTP/2] [1] [:authority: farnsworth.justagenericdomain.com]
  • [HTTP/2] [1] [:path: /.well-known/openid-configuration]
  • [HTTP/2] [1] [user-agent: curl/8.5.0]
  • [HTTP/2] [1] [accept: /]

GET /.well-known/openid-configuration HTTP/2 Host: farnsworth.justagenericdomain.com User-Agent: curl/8.5.0 Accept: /

< HTTP/2 200 < alt-svc: h3=":443"; ma=2592000 < cache-control: no-store < content-type: application/json < date: Mon, 09 Jun 2025 01:45:46 GMT < expires: Mon, 09 Jun 2025 00:45:46 GMT < pragma: no-cache < referrer-policy: strict-origin-when-cross-origin < set-cookie: __Host-zitadel.useragent=MTc0OTQzMzU0NnxRb3hkTHJNaGlrNEx1YmdBdGRtd2ZPX05leXZna0FGbkU1d0FjaDZpUzlRLVB0dm15MzIyNnBJbFVTdmRTUHhseEZ6MjFncktvSGxKaWlDcm9DaWlyQmRMWExkS2NBPT18VBa39jXKG2_8wKEo817T0AakG8cVFZbOhJXWxO9qRdQ=; Path=/; Max-Age=31536000; HttpOnly; Secure; SameSite=Lax < strict-transport-security: max-age=3600; includeSubDomains; preload < vary: Origin < vary: Cookie < x-content-type-options: nosniff < x-frame-options: SAMEORIGIN < x-robots-tag: none < x-xss-protection: 1; mode=block < content-length: 2274 < {"issuer":"https://farnsworth.justagenericdomain.com","authorization_endpoint":"https://farnsworth.justagenericdomain.com/oauth/v2/authorize","token_endpoint":"https://farnsworth.justagenericdomain.com/oauth/v2/token","introspection_endpoint":"https://farnsworth.justagenericdomain.com/oauth/v2/introspect","userinfo_endpoint":"https://farnsworth.justagenericdomain.com/oidc/v1/userinfo","revocation_endpoint":"https://farnsworth.justagenericdomain.com/oauth/v2/revoke","end_session_endpoint":"https://farnsworth.justagenericdomain.com/oidc/v1/end_session","device_authorization_endpoint":"https://farnsworth.justagenericdomain.com/oauth/v2/device_authorization","jwks_uri":"https://farnsworth.justagenericdomain.com/oauth/v2/keys","scopes_supported":["openid","profile","email","phone","address","offline_access"],"response_types_supported":["code","id_token","id_token token"],"response_modes_supported":["query","fragment","form_post"],"grant_types_supported":["authorization_code","implicit","refresh_token","client_credentials","urn:ietf:params:oauth:grant-type:jwt-bearer","urn:ietf:params:oauth:grant-type:device_code"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["RS256"],"request_object_signing_alg_values_supported":["RS256"],"token_endpoint_auth_methods_supported":["none","client_secret_basic","client_secret_post","private_key_jwt"],"token_endpoint_auth_signing_alg_values_supported":["RS256"],"revocation_endpoint_auth_methods_supported":["none","client_secret_basic","client_secret_post","private_key_jwt"],"revocation_endpoint_auth_signing_alg_values_supported":["RS256"],"introspection_endpoint_auth_methods_supported":["client_secret_basic","private_key_jwt"],"introspection_endpoint_auth_signing_alg_values_supported":["RS256"],"claims_supported":["sub","aud","exp","iat","iss","auth_time","nonce","acr","amr","c_hash","at_hash","act","scopes","client_id","azp","preferred_username","name","family_name","given_name","locale","email","email_verified","phone_number","phone_number_verified"],"code_challenge_methods_supported":["S256"],"ui_locales_supported":["bg","cs","de","en","es","fr","hu","id","it","ja","mk","nl","pl","pt","ru","sv","zh"],"request_parameter_supported":true,"request_uri_parameter_supported":false}

  • Connection #0 to host farnsworth.justagenericdomain.com left intact

here is the output of the grep of the management logs:

me="2025-06-08T18:47:37-07:00" level=warning msg="/opt/netbird/docker-compose.yml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion" management-1 | Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://farnsworth.justagenericdomain.com/.well-known/openid-configuration Get "https://farnsworth.justagenericdomain.com/.well-known/openid-configuration": dial tcp 50.193.208.182:443: connect: connection refused management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: management-1 | 2025-06-09T00:36:34Z WARN [context: SYSTEM] management/server/account.go:249: failed to parse peer update interval, using default value 1ms: strconv.Atoi: parsing "": invalid syntax management-1 | 2025-06-09T00:36:34Z WARN management/server/token_mgr.go:61: TURN credentials TTL is not set or invalid, using default value 12h0m0s management-1 | 2025-06-09T00:36:34Z WARN [context: SYSTEM] management/cmd/management.go:222: TrustedPeers are configured to default value '0.0.0.0/0', '::/0'. This allows connection IP spoofing. management-1 | 2025-06-09T00:37:40Z WARN [requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428, context: HTTP] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [accountID: cui0e08rel3s739ad29g, userID: 305860109269991428, context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [userID: 305860109269991428, context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428, context: HTTP] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [userID: 305860109269991428, context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [userID: 305860109269991428, context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428, context: HTTP] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [userID: 305860109269991428, context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428, context: HTTP] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [userID: 305860109269991428, context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1754782792 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1885021493 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1885021493 management-1 | 2025-06-09T00:37:40Z WARN [userID: 305860109269991428, context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g] management/server/event.go:97: failed to resolve user info for initiator: 1885021493 management-1 | 2025-06-09T00:37:40Z WARN [accountID: cui0e08rel3s739ad29g, userID: 305860109269991428, context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20] management/server/event.go:97: failed to resolve user info for initiator: 1885021493 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 1885021493 management-1 | 2025-06-09T00:37:40Z WARN [userID: 305860109269991428, context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g] management/server/event.go:97: failed to resolve user info for initiator: 44451462 management-1 | 2025-06-09T00:37:40Z WARN [context: HTTP, requestID: 6a5913ea-93c6-4487-ae55-f638487e8c20, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/event.go:97: failed to resolve user info for initiator: 3383495720 management-1 | 2025-06-09T00:37:59Z ERRO [context: HTTP, requestID: 4e9e0e90-0e10-4619-991b-956de2b4fb9c, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/http/util/util.go:85: got a handler error: peer not found: cukgjggrel3s73ah9cig management-1 | 2025-06-09T00:37:59Z ERRO [context: HTTP, requestID: 4e9e0e90-0e10-4619-991b-956de2b4fb9c] management/server/telemetry/http_api_metrics.go:189: HTTP response 4e9e0e90-0e10-4619-991b-956de2b4fb9c: GET /api/peers/cukgjggrel3s73ah9cig status 404 management-1 | 2025-06-09T00:38:04Z ERRO [requestID: 40ef2bcb-bc23-43f2-ab38-ff10b4a6264a, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428, context: HTTP] management/server/http/util/util.go:85: got a handler error: peer not found: cukgjggrel3s73ah9cig management-1 | 2025-06-09T00:38:04Z ERRO [context: HTTP, requestID: 40ef2bcb-bc23-43f2-ab38-ff10b4a6264a] management/server/telemetry/http_api_metrics.go:189: HTTP response 40ef2bcb-bc23-43f2-ab38-ff10b4a6264a: GET /api/peers/cukgjggrel3s73ah9cig status 404 management-1 | 2025-06-09T01:43:06Z ERRO [context: HTTP, requestID: d6ef78d0-e31a-4552-a457-619f51a140f4, accountID: cui0e08rel3s739ad29g, userID: 305860109269991428] management/server/http/util/util.go:85: got a handler error: peer not found: cukgjggrel3s73ah9cig management-1 | 2025-06-09T01:43:06Z ERRO [requestID: d6ef78d0-e31a-4552-a457-619f51a140f4, context: HTTP] management/server/telemetry/http_api_metrics.go:189: HTTP response d6ef78d0-e31a-4552-a457-619f51a140f4: GET /api/peers/cukgjggrel3s73ah9cig status 404

scroguard avatar Jun 09 '25 01:06 scroguard

i ended up having to nuke and completely reinstall the system, so i won't be able to provide any further insight/logs/etc. past what has already been provided.

if this randomly occurs again, it will be enough to stop me from using netbird and having to look at other solutions as a complete rebuild of everything is not something i care to have to repeat.

scroguard avatar Jun 17 '25 22:06 scroguard