netbirdio
Management Service login issue
Hello, I started using Netbird yesterday on my linux machine. It was working fine, but once i restarted my machine it displays the message "2023-08-03T10:19:37+01:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 1.098619925s due to error rpc error: code = Unknown desc = getting pkce authorization flow info failed with error: context deadline exceeded" when i run "sudo netbird up" even with my setup-key. I tried uninstall/reinstall but nothing worked. Can you help please ?
Steps to reproduce the behavior:
- Install Netbird
- sudo netbird up
- restart machine
- sudo netbird up
- 2023-08-03T10:19:37+01:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 1.098619925s due to error rpc error: code = Unknown desc = getting pkce authorization flow info failed with error: context deadline exceeded
The expected behavior is to have the same as yesterday when i installed it. I expected it to display "connected" after i run the command "sudo netbird up".
NetBird status -d output: Daemon status: LoginFailed
Run UP command to log in with SSO (interactive login):
netbird up
If you are running a self-hosted version and no SSO provider has been configured in your Management Server, you can use a setup-key:
netbird up --management-url <YOUR_MANAGEMENT_URL> --setup-key <YOUR_SETUP_KEY>
More info: https://www.netbird.io/docs/overview/setup-keys
Screenshots
Thanks for help
Hi. Almost identical similar issue here. Just to add some additional context of my case that may be helpful:
- For me, errors with the connection to Management Server start as soon as seconds after the registration as a peer. (see image attached)
- The only instance in which I am seeing this issue myself is in a docker Ubuntu container, Netbird client is working as expected in all other machines in which I have installed it. The container I am mentioning is inside the same server as the management service and it is running a client because I wanted to use as a routing peer in order to not have to create a separate machine.
I have been trying to debug the error but I have had no luck so far, any help or discoveries on the matter would be greatly appreciated if shared.
Thanks!
Could you please check if there are any error logs available at /var/log/netbird/client.log and /var/log/netbird/netbird.err?
Here you go, these are the logs from the client that is having the issues mentioned. As far as I have seen, the other machines (clients that work properly and the management server, don't have these logs).
Checking timestamps, the line that aligns perfectly with the moment in which the issues have started is this one:
2023-08-03T10:50:03Z ERRO client/internal/wgproxy/factory_linux.go:15: failed to initialize ebpf proxy: failed to set memlock rlimit: operation not permitted
@gmatasba @Sxndxr the issue was fixed in v0.22.2, can you test on your end?
I had a similar issue with netbird client v0.21.8:
netbird up --management-url <YOUR_MANAGEMENT_URL> --setup-key <YOUR_SETUP_KEY>
logs
2023-08-14T16:01:13+02:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 936.119771ms due to error rpc error: code = DeadlineExceeded desc = context deadline exceeded 2023-08-14T16:01:24+02:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 2.102444321s due to error rpc error: code = DeadlineExceeded desc = context deadline exceeded Error: login backoff cycle failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Updating the client to v0.22.4 fixed it for me. Yay.
I have exactly the same Problem on one (I have several) Manjaro Linux computer with the latest netbird Client from AUR (not docker).
Tried to remove the package and reinstalled it, but had no luck so far.
Any idea what causes this problem?
@mlsmaycon I personally solved the issue by creating a very small separate machine in order to use it as a routing peer and no longer have the set-up I described earlier. I will try to replicate it, test with the new version and then share the results here.
Since today a second machine fails to connect. When I execute "netbird up", I get:
2023-08-22T14:52:40+02:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 502.410121ms due to error rpc error: code = Unknown desc = getting pkce authorization flow info failed with error: context deadline exceeded Error: login backoff cycle failed: rpc error: code = Unknown desc = getting pkce authorization flow info failed with error: context deadline exceeded
Daemon version: 0.22.6 CLI version: 0.22.6
What is going on?
Getting the same error on v0.22.7 with Debian 12 LXC. Interesting enough I was able to get it connected but after restarting the container that's when the errors started popping up and even started locking up the LXC container every 20 secs and dropping my SSH connection.
I think you have to update Netbird. I guess that the bug has been fixed in the new version.
Le mer. 30 août 2023 à 05:03, Trevin Corkery @.***> a écrit :
Getting the same error on v0.22.7 with Debian 12 LXC. Interesting enough I was able to get it connected but after restarting the container that's when the errors started popping up and even started locking up the LXC container every 20 secs and dropping my SSH connection.
— Reply to this email directly, view it on GitHub https://github.com/netbirdio/netbird/issues/1048#issuecomment-1698418127, or unsubscribe https://github.com/notifications/unsubscribe-auth/AUQXX7WG3S23PMVNL6LVCNTXX2URVANCNFSM6AAAAAA3CPJA34 . You are receiving this because you were mentioned.Message ID: @.***>
Same here with newest version:
Exactly same when --management-url provided
The issue might be a problem with gRPC connectivity. Could you run the agent in the foreground with the following steps and share the output?
First, you need to stop the current agent:
sudo netbird service stop
Then run:
sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info netbird up -F -l debug'
@mlsmaycon Here is the log that I get. I just try to install and run netbird on my 2 machines. The desktop runs normally while the laptop encounters this issue. Both have the same network configurations.
// Edit: My bad. Turn out on my laptop, I didn't link stub-resolv.conf to /etc/resolv.conf. It's now working normally.
I still have this issue with the latest releases.
sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info netbird up -F -l debug'
2023-09-19T09:17:12+02:00 DEBG client/internal/login.go:93: connecting to the Management service https://api.wiretrustee.com:443
2023/09/19 09:17:12 INFO: [core] [Channel #1] Channel created
2023/09/19 09:17:12 INFO: [core] [Channel #1] original dial target is: "api.wiretrustee.com:443"
2023/09/19 09:17:12 INFO: [core] [Channel #1] parsed dial target is: {Scheme:api.wiretrustee.com Authority: URL:{Scheme:api.wiretrustee.com Opaque:443 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/19 09:17:12 INFO: [core] [Channel #1] fallback to scheme "passthrough"
2023/09/19 09:17:12 INFO: [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/api.wiretrustee.com:443 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/19 09:17:12 INFO: [core] [Channel #1] Channel authority set to "api.wiretrustee.com:443"
2023/09/19 09:17:12 INFO: [core] [Channel #1] Resolver state updated: {
"Addresses": [
{
"Addr": "api.wiretrustee.com:443",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/19 09:17:12 INFO: [core] [Channel #1] Channel switches to new LB policy "pick_first"
2023/09/19 09:17:12 INFO: [core] [Channel #1 SubChannel #2] Subchannel created
2023/09/19 09:17:12 INFO: [core] [Channel #1] Channel Connectivity change to CONNECTING
2023/09/19 09:17:12 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING
2023/09/19 09:17:12 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "api.wiretrustee.com:443" to connect
2023/09/19 09:17:12 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0006da0a8, {CONNECTING
Any ideas?
I updated netbird client from 0.22.7 to 0.23.2 on my CentOS 7.9 host and have got the same issue:
2023-09-22T18:43:01+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-22T18:43:02+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023-09-22T18:43:02+03:00 ERRO client/server/server.go:124: failed login: context deadline exceeded
At the same time there isn't this issue on fedora 37 with 0.23.2
UPDATE: reinstall netbird client 0.22.7 which worked well before and have got the same issue DEBUG:
sudo bash -c 'GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info netbird up -F -l debug'
2023-09-25T10:46:57+03:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.hidden.tech:33073
2023/09/25 10:46:57 INFO: [core] [Channel #1] Channel created
2023/09/25 10:46:57 INFO: [core] [Channel #1] original dial target is: "netbird.hidden.tech:33073"
2023/09/25 10:46:57 INFO: [core] [Channel #1] parsed dial target is: {Scheme:netbird.hidden.tech Authority: URL:{Scheme:netbird.hidden.tech Opaque:33073 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:46:57 INFO: [core] [Channel #1] fallback to scheme "passthrough"
2023/09/25 10:46:57 INFO: [core] [Channel #1] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/netbird.hidden.tech:33073 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:46:57 INFO: [core] [Channel #1] Channel authority set to "netbird.hidden.tech:33073"
2023/09/25 10:46:57 INFO: [core] [Channel #1] Resolver state updated: {
"Addresses": [
{
"Addr": "netbird.hidden.tech:33073",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/25 10:46:57 INFO: [core] [Channel #1] Channel switches to new LB policy "pick_first"
2023/09/25 10:46:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel created
2023/09/25 10:46:57 INFO: [core] [Channel #1] Channel Connectivity change to CONNECTING
2023/09/25 10:46:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING
2023/09/25 10:46:57 INFO: [core] [Channel #1 SubChannel #2] Subchannel picks a new address "netbird.hidden.tech:33073" to connect
2023/09/25 10:46:57 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc000010ba0, {CONNECTING <nil>}
2023/09/25 10:47:02 INFO: [core] [Channel #1] Channel Connectivity change to SHUTDOWN
2023/09/25 10:47:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to SHUTDOWN
2023/09/25 10:47:02 INFO: [core] [Channel #1 SubChannel #2] Subchannel deleted
2023/09/25 10:47:02 INFO: [core] [Channel #1] Channel deleted
2023-09-25T10:47:02+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-25T10:47:02+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023/09/25 10:47:02 INFO: [core] Creating new client transport to "{\n \"Addr\": \"netbird.hidden.tech:33073\",\n \"ServerName\": \"netbird.hidden.tech:33073\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}": connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023/09/25 10:47:02 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "netbird.hidden.tech:33073",
"ServerName": "netbird.hidden.tech:33073",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023-09-25T10:47:02+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 781.98239ms due to error context deadline exceeded
2023-09-25T10:47:03+03:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.hidden.tech:33073
2023/09/25 10:47:03 INFO: [core] [Channel #3] Channel created
2023/09/25 10:47:03 INFO: [core] [Channel #3] original dial target is: "netbird.hidden.tech:33073"
2023/09/25 10:47:03 INFO: [core] [Channel #3] parsed dial target is: {Scheme:netbird.hidden.tech Authority: URL:{Scheme:netbird.hidden.tech Opaque:33073 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:03 INFO: [core] [Channel #3] fallback to scheme "passthrough"
2023/09/25 10:47:03 INFO: [core] [Channel #3] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/netbird.hidden.tech:33073 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:03 INFO: [core] [Channel #3] Channel authority set to "netbird.hidden.tech:33073"
2023/09/25 10:47:03 INFO: [core] [Channel #3] Resolver state updated: {
"Addresses": [
{
"Addr": "netbird.hidden.tech:33073",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/25 10:47:03 INFO: [core] [Channel #3] Channel switches to new LB policy "pick_first"
2023/09/25 10:47:03 INFO: [core] [Channel #3 SubChannel #4] Subchannel created
2023/09/25 10:47:03 INFO: [core] [Channel #3] Channel Connectivity change to CONNECTING
2023/09/25 10:47:03 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to CONNECTING
2023/09/25 10:47:03 INFO: [core] [Channel #3 SubChannel #4] Subchannel picks a new address "netbird.hidden.tech:33073" to connect
2023/09/25 10:47:03 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0001aa078, {CONNECTING <nil>}
2023/09/25 10:47:08 INFO: [core] [Channel #3] Channel Connectivity change to SHUTDOWN
2023/09/25 10:47:08 INFO: [core] [Channel #3 SubChannel #4] Subchannel Connectivity change to SHUTDOWN
2023/09/25 10:47:08 INFO: [core] [Channel #3 SubChannel #4] Subchannel deleted
2023/09/25 10:47:08 INFO: [core] [Channel #3] Channel deleted
2023-09-25T10:47:08+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-25T10:47:08+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023/09/25 10:47:08 INFO: [core] Creating new client transport to "{\n \"Addr\": \"netbird.hidden.tech:33073\",\n \"ServerName\": \"netbird.hidden.tech:33073\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}": connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023-09-25T10:47:08+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 1.993228132s due to error context deadline exceeded
2023/09/25 10:47:08 WARNING: [core] [Channel #3 SubChannel #4] grpc: addrConn.createTransport failed to connect to {
"Addr": "netbird.hidden.tech:33073",
"ServerName": "netbird.hidden.tech:33073",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023-09-25T10:47:10+03:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.hidden.tech:33073
2023/09/25 10:47:10 INFO: [core] [Channel #5] Channel created
2023/09/25 10:47:10 INFO: [core] [Channel #5] original dial target is: "netbird.hidden.tech:33073"
2023/09/25 10:47:10 INFO: [core] [Channel #5] parsed dial target is: {Scheme:netbird.hidden.tech Authority: URL:{Scheme:netbird.hidden.tech Opaque:33073 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:10 INFO: [core] [Channel #5] fallback to scheme "passthrough"
2023/09/25 10:47:10 INFO: [core] [Channel #5] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/netbird.hidden.tech:33073 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:10 INFO: [core] [Channel #5] Channel authority set to "netbird.hidden.tech:33073"
2023/09/25 10:47:10 INFO: [core] [Channel #5] Resolver state updated: {
"Addresses": [
{
"Addr": "netbird.hidden.tech:33073",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/25 10:47:10 INFO: [core] [Channel #5] Channel switches to new LB policy "pick_first"
2023/09/25 10:47:10 INFO: [core] [Channel #5 SubChannel #6] Subchannel created
2023/09/25 10:47:10 INFO: [core] [Channel #5] Channel Connectivity change to CONNECTING
2023/09/25 10:47:10 INFO: [core] [Channel #5 SubChannel #6] Subchannel Connectivity change to CONNECTING
2023/09/25 10:47:10 INFO: [core] [Channel #5 SubChannel #6] Subchannel picks a new address "netbird.hidden.tech:33073" to connect
2023/09/25 10:47:10 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0001aa408, {CONNECTING <nil>}
2023/09/25 10:47:15 INFO: [core] [Channel #5] Channel Connectivity change to SHUTDOWN
2023/09/25 10:47:15 INFO: [core] [Channel #5 SubChannel #6] Subchannel Connectivity change to SHUTDOWN
2023/09/25 10:47:15 INFO: [core] [Channel #5 SubChannel #6] Subchannel deleted
2023/09/25 10:47:15 INFO: [core] [Channel #5] Channel deleted
2023-09-25T10:47:15+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-25T10:47:15+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023/09/25 10:47:15 INFO: [core] Creating new client transport to "{\n \"Addr\": \"netbird.hidden.tech:33073\",\n \"ServerName\": \"netbird.hidden.tech:33073\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}": connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023/09/25 10:47:15 WARNING: [core] [Channel #5 SubChannel #6] grpc: addrConn.createTransport failed to connect to {
"Addr": "netbird.hidden.tech:33073",
"ServerName": "netbird.hidden.tech:33073",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023-09-25T10:47:15+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 2.225361164s due to error context deadline exceeded
2023-09-25T10:47:17+03:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.hidden.tech:33073
2023/09/25 10:47:17 INFO: [core] [Channel #7] Channel created
2023/09/25 10:47:17 INFO: [core] [Channel #7] original dial target is: "netbird.hidden.tech:33073"
2023/09/25 10:47:17 INFO: [core] [Channel #7] parsed dial target is: {Scheme:netbird.hidden.tech Authority: URL:{Scheme:netbird.hidden.tech Opaque:33073 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:17 INFO: [core] [Channel #7] fallback to scheme "passthrough"
2023/09/25 10:47:17 INFO: [core] [Channel #7] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/netbird.hidden.tech:33073 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:17 INFO: [core] [Channel #7] Channel authority set to "netbird.hidden.tech:33073"
2023/09/25 10:47:17 INFO: [core] [Channel #7] Resolver state updated: {
"Addresses": [
{
"Addr": "netbird.hidden.tech:33073",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/25 10:47:17 INFO: [core] [Channel #7] Channel switches to new LB policy "pick_first"
2023/09/25 10:47:17 INFO: [core] [Channel #7 SubChannel #8] Subchannel created
2023/09/25 10:47:17 INFO: [core] [Channel #7] Channel Connectivity change to CONNECTING
2023/09/25 10:47:17 INFO: [core] [Channel #7 SubChannel #8] Subchannel Connectivity change to CONNECTING
2023/09/25 10:47:17 INFO: [core] [Channel #7 SubChannel #8] Subchannel picks a new address "netbird.hidden.tech:33073" to connect
2023/09/25 10:47:17 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0006660a8, {CONNECTING <nil>}
2023/09/25 10:47:22 INFO: [core] [Channel #7] Channel Connectivity change to SHUTDOWN
2023/09/25 10:47:22 INFO: [core] [Channel #7 SubChannel #8] Subchannel Connectivity change to SHUTDOWN
2023/09/25 10:47:22 INFO: [core] [Channel #7 SubChannel #8] Subchannel deleted
2023/09/25 10:47:22 INFO: [core] [Channel #7] Channel deleted
2023-09-25T10:47:22+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-25T10:47:22+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023-09-25T10:47:22+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 1.731637762s due to error context deadline exceeded
2023/09/25 10:47:22 INFO: [core] Creating new client transport to "{\n \"Addr\": \"netbird.hidden.tech:33073\",\n \"ServerName\": \"netbird.hidden.tech:33073\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}": connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023/09/25 10:47:22 WARNING: [core] [Channel #7 SubChannel #8] grpc: addrConn.createTransport failed to connect to {
"Addr": "netbird.hidden.tech:33073",
"ServerName": "netbird.hidden.tech:33073",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing: dial tcp: lookup netbird.hidden.tech: operation was canceled"
2023-09-25T10:47:24+03:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.hidden.tech:33073
2023/09/25 10:47:24 INFO: [core] [Channel #9] Channel created
2023/09/25 10:47:24 INFO: [core] [Channel #9] original dial target is: "netbird.hidden.tech:33073"
2023/09/25 10:47:24 INFO: [core] [Channel #9] parsed dial target is: {Scheme:netbird.hidden.tech Authority: URL:{Scheme:netbird.hidden.tech Opaque:33073 User: Host: Path: RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:24 INFO: [core] [Channel #9] fallback to scheme "passthrough"
2023/09/25 10:47:24 INFO: [core] [Channel #9] parsed dial target is: {Scheme:passthrough Authority: URL:{Scheme:passthrough Opaque: User: Host: Path:/netbird.hidden.tech:33073 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}
2023/09/25 10:47:24 INFO: [core] [Channel #9] Channel authority set to "netbird.hidden.tech:33073"
2023/09/25 10:47:24 INFO: [core] [Channel #9] Resolver state updated: {
"Addresses": [
{
"Addr": "netbird.hidden.tech:33073",
"ServerName": "",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}
],
"ServiceConfig": null,
"Attributes": null
} (resolver returned new addresses)
2023/09/25 10:47:24 INFO: [core] [Channel #9] Channel switches to new LB policy "pick_first"
2023/09/25 10:47:24 INFO: [core] [Channel #9 SubChannel #10] Subchannel created
2023/09/25 10:47:24 INFO: [core] [Channel #9] Channel Connectivity change to CONNECTING
2023/09/25 10:47:24 INFO: [core] [Channel #9 SubChannel #10] Subchannel Connectivity change to CONNECTING
2023/09/25 10:47:24 INFO: [core] [Channel #9 SubChannel #10] Subchannel picks a new address "netbird.hidden.tech:33073" to connect
2023/09/25 10:47:24 INFO: [core] pickfirstBalancer: UpdateSubConnState: 0xc0006f8090, {CONNECTING <nil>}
2023/09/25 10:47:29 INFO: [core] [Channel #9] Channel Connectivity change to SHUTDOWN
2023/09/25 10:47:29 INFO: [core] [Channel #9 SubChannel #10] Subchannel Connectivity change to SHUTDOWN
2023/09/25 10:47:29 INFO: [core] [Channel #9 SubChannel #10] Subchannel deleted
2023/09/25 10:47:29 INFO: [core] [Channel #9] Channel deleted
2023-09-25T10:47:29+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-09-25T10:47:29+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
Error: foreground login failed: backoff cycle failed: context deadline exceeded
Any news? This issue is very annoying and a deal breaker for me, if no solution is around the corner...
Could you please upgrade to the most recent version, v0.23.6? After the update, follow these steps:
- Execute
sudo netbird service stop - Run
sudo netbird service uninstall - Then, proceed with
sudo netbird service install - Finally, initiate
sudo netbird service startand test the client's connection withnetbird up
There is no v0.23.6 available for my centos - v0.23.4 is the latest available.
I've tried your solution but result the same:
Downloading packages:
netbird_0.23.4_linux_amd64.rpm | 8.4 MB 00:00:05
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : netbird-0.23.4-1.x86_64 1/1 Post Install of an clean install
Netbird service has been installed
Netbird service has been started
Verifying : netbird-0.23.4-1.x86_64 1/1
Installed:
netbird.x86_64 0:0.23.4-1
Complete!
:~$ sudo netbird service stop
Netbird service has been stopped
:~$ sudo netbird service uninstall
Netbird has been uninstalled
:~$ sudo netbird service install
Netbird service has been installed
:~$ sudo netbird service start
Netbird service has been started
:~$ sudo netbird up --management-url https://netbird.hidden.tech:33073 --setup-key 'hidden'
Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed connecting to Management Service : context deadline exceeded
:~$ sudo netbird down
sudo netbird up --management-url https://netbird.hidden.tech:33073 --setup-key 'hidden'
2023-10-04T10:39:55+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 890.429447ms due to error rpc error: code = DeadlineExceeded desc = context deadline exceeded
2023-10-04T10:40:06+03:00 WARN client/cmd/root.go:187: retrying Login to the Management service in 1.12732371s due to error rpc error: code = DeadlineExceeded desc = context deadline exceeded
Error: login backoff cycle failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
:~$ tail /var/log/netbird/client.log
2023-10-04T10:40:01+03:00 ERRO client/server/server.go:124: failed login: context deadline exceeded
2023-10-04T10:40:06+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-10-04T10:40:06+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023-10-04T10:40:06+03:00 ERRO client/server/server.go:124: failed login: context deadline exceeded
2023-10-04T10:40:12+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-10-04T10:40:12+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023-10-04T10:40:12+03:00 ERRO client/server/server.go:124: failed login: context deadline exceeded
2023-10-04T10:40:17+03:00 ERRO management/client/grpc.go:64: failed creating connection to Management Service context deadline exceeded
2023-10-04T10:40:17+03:00 ERRO client/internal/login.go:96: failed connecting to the Management service https://netbird.hidden.tech:33073 context deadline exceeded
2023-10-04T10:40:17+03:00 ERRO client/server/server.go:124: failed login: context deadline exceeded
:~$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Any other suggestion?
~ > sudo netbird service stop
Netbird service has been stopped
~ > sudo netbird service uninstall
Netbird has been uninstalled
~ > sudo netbird service install
Netbird service has been installed
~ > sudo netbird service start
Netbird service has been started
~ > netbird up
Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed connecting to Management Service : context deadline exceeded
~ > netbird version
0.23.6
@kdre , @devopskupryk. To streamline the troubleshooting process and address your issue more effectively, we kindly request that you join our Slack or Matrix.
I also stumbled across this kind of error today. For me the solution was to adjust my /etc/resolv.conf. there were two entries for nameserver + search domain. after removing the second (original host) entry and keeping the one created by netbird, a "netbird up" worked again.
it seems that "netbird up" failed to resolve its required login servers.
I have this problem:
Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed while getting Management Service public key: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); malformed header: missing HTTP content-type
I also stumbled across this kind of error today. For me the solution was to adjust my /etc/resolv.conf. there were two entries for nameserver + search domain. after removing the second (original host) entry and keeping the one created by netbird, a "netbird up" worked again.
it seems that "netbird up" failed to resolve its required login servers.
This also fixed it for me, apparently adding a custom DNS nameserver in netbird that's only accessible via the VPN causes this.
In my case, it was because I changed my Netbird host url (like... old.example.com → new.example.io).
Fix was to delete (or move to /tmp) the config.json in /etc/netbird/config.json,
and ran sudo netbird service uninstall. Not sure if which handled the issue though
Just want to bring attention to this issue again. We had the same problem today with one of our servers. Server rebooted due to a failure (non Netbird related) and did not show up in Netbird as online. After running the command netbird up manually, we got the same error message as some of you above:
Error: status failed: failed connecting to Management Service : context deadline exceeded
We figured out that Netbird replaces or adds its own Nameserver (if configured) and search into the /etc/resolv.conf. The order was Netbird-configured-search, Netbird-configured-nameserver and then our two nameservers. We also noticed problems with the general DNS resolution to e.g. google.de. Our Netbird instance address resolves via a public DNS and therefore this setup won't work.
After removing all entries except our two public nameservers and waiting a few seconds, we could successfully connect to our Netbird instance with netbird up. This, obviously, overwrote the /etc/resolv.conf again. This will likely result in the same scenario if the machine goes down again the next time.
Is there any way to avoid this problem here?
Just want to bring attention to this issue again. We had the same problem today with one of our servers. Server rebooted due to a failure (non Netbird related) and did not show up in Netbird as online. After running the command
netbird upmanually, we got the same error message as some of you above:Error: status failed: failed connecting to Management Service : context deadline exceededWe figured out that Netbird replaces or adds its own Nameserver (if configured) and search into the
/etc/resolv.conf. The order was Netbird-configured-search, Netbird-configured-nameserver and then our two nameservers. We also noticed problems with the general DNS resolution to e.g. google.de. Our Netbird instance address resolves via a public DNS and therefore this setup won't work.After removing all entries except our two public nameservers and waiting a few seconds, we could successfully connect to our Netbird instance with
netbird up. This, obviously, overwrote the/etc/resolv.confagain. This will likely result in the same scenario if the machine goes down again the next time.Is there any way to avoid this problem here?
I found the same behavior. After removing the servers from resolv.conf, netbird connected.
I have this problem:
Error: unable to get daemon status: rpc error: code = FailedPrecondition desc = failed while getting Management Service public key: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); malformed header: missing HTTP content-type
this is the error I also have. Where you able to resolve your issue?
