Stuck on loading dashboard

[MichalMarchewka]

trafficstars

Describe the problem

A Netbird is stuck on loading dashboard.

To Reproduce

Steps to reproduce the behavior:

1. Install and setup Netbird following this guide
2. Try to sign in to Netbird. After putting credentials and MFA Netbird stucks.

Expected behavior

Netbirds successfully loads and https://netbird.REDACTED.net/peers content is reached

Are you using NetBird Cloud?

Self-hosted

NetBird version

0.28.3

NetBird status -d output:

If applicable, add the `netbird status -d' command output.

Screenshots

Additional context

• Netbird is installed on 1 CPU and 2 GM RAM VPS
• docker compose logs contains some errors:

zitadel-1     | time="2024-07-01T21:00:28Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=auth.users3
zitadel-1     | time="2024-07-01T21:00:37Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=auth.tokens
zitadel-1     | time="2024-07-01T21:00:38Z" level=warning msg="instance by host" caller="/home/runner/work/zitadel/zitadel/internal/query/instance.go:210" domain=netbird.REDACTED.net error="failed to connect to `host=zdb user=zitadel database=zitadel`: failed to receive message (context canceled)" host=netbird.REDACTED.net
zitadel-1     | time="2024-07-01T21:00:38Z" level=error msg="unable to set instance" caller="/home/runner/work/zitadel/zitadel/internal/api/http/middleware/instance_interceptor.go:60" error="unable to get instance by host netbird.REDACTED.net: failed to connect to `host=zdb user=zitadel database=zitadel`: failed to receive message (context canceled)" externalDomain=netbird.REDACTED.net origin="https://netbird.REDACTED.net"
zitadel-1     | time="2024-07-01T21:00:38Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.apps7
zitadel-1     | time="2024-07-01T21:00:51Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.restrictions2
zitadel-1     | time="2024-07-01T21:00:51Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.idp_login_policy_links5
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.org_domains2
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.password_complexity_policies2
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.actions3
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.system_features
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:414" iteration=0 projection=projections.idp_user_links3
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="unable to query current state" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/state.go:70" error="ERROR: could not obtain lock on row in relation \"current_states\" (SQLSTATE 55P03)" projection=auth.user_sessions
zitadel-1     | time="2024-07-01T21:00:52Z" level=debug msg="state already locked" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:462" projection=auth.user_sessions

[Jadefalkner]

sounds similar to the problem i faced? maybe it helps

https://github.com/netbirdio/netbird/issues/1699

[leoboyerbx]

Same problem here

[snoopckuu]

Having same issue with Google iDP in Safari. Dashboard won't load and after sometime redirects back to Google Auth window. However Works in Google Chrome and Edge.

[mlsmaycon]

@MichalMarchewka can you share the caddy logs?

[mlsmaycon]

Same problem here

with our quick start guide too? if so, can you share the caddy logs?

[mlsmaycon]

Having same issue with Google iDP in Safari. Dashboard won't load and after sometime redirects back to Google Auth window. However Works in Google Chrome and Edge.

thanks for reporting, we will validate the issue with Google as IdP. In the mean time, can you check if https://app.netbird.io works well for you?

[snoopckuu]

Having same issue with Google iDP in Safari. Dashboard won't load and after sometime redirects back to Google Auth window. However Works in Google Chrome and Edge.

thanks for reporting, we will validate the issue with Google as IdP. In the mean time, can you check if https://app.netbird.io works well for you?

Yep, works well.

[MichalMarchewka]

@mlsmaycon Please find logs attached caddy_logs.txt

[joshuahigginson1]

Hi @mlsmaycon , it appears that we've got the same issue after upgrading from v2.3.0 to v2.4.1 of the Netbird Dashboard. Both Chrome and Safari - I was never convinced that we had our auth configured correctly.

Looks like an issue with the change in the Dashboard useRedirect hook.

--- v2.3.0

Logging in or refreshing the page would result in a 100ms ~ 2sec wait for the console to show the Google Accounts page. This will always appear after a 'timeout' appears in the Chrome console.

After authenticating to Google, we see the following logs:

GET https://<our-domain>/auth 404 (Not Found)
Checking to see if there is an authorization response to be delivered.
Potential authorization request  https://<our-domain>/auth ...
Delivering authorization response

Peers page then loads.

Refreshing the page restarts this cycle and loads the Google Accounts page again.

Note: Chrome and Safari both block third-party cookies which is probably why we have to re-authenticate every page refresh.

---v2.4.1

Logging in or refreshing the page would result in a 100ms ~ 2sec wait for the console to show the Google Accounts page. This will always appear after a 'timeout' appears in the Chrome console. The difference between two versions is that we have 36 counts of 'Third Party Cookie will be Blocked' before Google Accounts page appears.

After authenticating to Google, we see the following logs:

GET https://<our-domain>/auth 404 (Not Found)
Checking to see if there is an authorization response to be delivered.
Potential authorization request  https://<our-domain>/auth ...
Delivering authorization response
** GET https://<our-domain>/auth.txt 404 (Not Found) **

Peers page never loads. Eventually, Netbird takes us to the Google Accounts page again.

[axlroden]

I have the same issue after updating to latest image, and running a previously working netbird with google idp.

For some reason it works in firefox, but not in chrome or safari.

Downgrading to v2.3.0 dashboard resolves the issue..

[mlsmaycon]

@heisbrot can you have a look at this issue?

[heisbrot]

Hey @MichalMarchewka ,

can you check if your SSL is properly configured? I see some certificate errors in the logs. Was you domain publicly accessible before you started with the getting started guide?

@joshuahigginson1 @axlroden @snoopckuu Do you have some logs? Or maybe a HAR file for the network requests. You can also try clearing the cache and cookies of your browser and check if the issue persists. (Or opening incognito) Does rebuilding the container fix the issue? docker compose up -d --force-recreate dashboard

[ylluminate]

I had the same problem, but it seems to be resolved by adjusting the /etc/hosts file. Here's what worked for me:

# /etc/hosts
127.0.0.1       localhost

Initially, I had other hostnames defined before localhost, but this setup caused issues with Docker containers accessing the identity provider configuration. Simplifying the /etc/hosts file to include only localhost for IPv4 resolved the problem.

As we're all probably aware, the /etc/hosts file is used to map IP addresses to hostnames. It's common practice to include multiple hostnames on a single line, such as 127.0.1.1 hostname.domain.tld hostname localhost, to associate multiple hostnames with an IP address. However, this configuration appears to interfere with Docker's DNS resolution inside containers.

In this case, the management container was repeatedly failing to fetch the OIDC configuration due to connection refusals on 127.0.1.1. Changing the /etc/hosts file to the above did resolve this...

If this is what's affecting others as well, and to perhaps avoid similar issues in the future, it might be beneficial to document this behavior as a known issue or recommend best practices for /etc/hosts configurations in the setup guides. Additionally, it could be helpful to investigate why Docker's DNS resolution behaves differently with multiple hostnames and whether there are configuration changes that can make it more robust to different /etc/hosts setups.

I'm not a big Docker fan myself, so I'm not terribly inclined to dig into this any further myself.

[MichalMarchewka]

Hey @heisbrot Yes, I have now rebuilt it from scratch. Unfortunately, it still doesn't work. Is there a chance to install Netbird without Docker?

Editing /etc/hosts per ylluminate instruction doesn't work for me.

[ylluminate]

@MichalMarchewka that is unfortunate to hear. Had hoped that would work for you. I do agree though, having a non-Docker install path would be good. Docker is such a headache.

[axlroden]

management.txt dashboard.txt 2.4.0.har.txt Logs of management and dashboard, and har file on 2.4.0 dashboard.

I tried doing force-recreate as well. Tried without cache on multiple browsers. Only firefox ends up on a peer list, after several reloads. Others either keep refreshing indefinately, or shows the google accounts login screen.

[heisbrot]

Hey @axlroden

Thank you for the logs. Can you try with the following image netbirdio/dashboard:pr-403 (instead of latest) and see if the problem persists?

[jeehoonkang]

@heisbrot I had the same problem, and netbirdio/dashboard:pr-403 fixed the issue. Thanks!

[joshuahigginson1]

@heisbrot Thanks you Eduard, PR-403 has solved our issue.

[snoopckuu]

Can we merge the fix into the main branch?

[axlroden]

can confirm netbirdio/dashboard:pr-403 solved the issue.

[MichalMarchewka]

Amazing stuff, it's working now!

Good job, thank you.

[TheKayneGame]

This issue has not been resolved for me with the PR.

My setup contains Traefik and followed this tutorial from JimsGarage's video.

i am facing the same /peers screen loop.

See netbird-logs.txt for my logs.

I am running it on a local machine and externally its connected through cloudflare.

[git-day]

@heisbrot I am also experiencing the same problem as @MichalMarchewka where the login works as expected, but the dashboard never loads.

My setup is new and have followed the advanced guide here https://docs.netbird.io/selfhosted/selfhosted-guide#requirements. The chosen Idp is Zitadel and have followed the guide here on setup https://docs.netbird.io/selfhosted/identity-providers#zitadel.

I've noticed that there are a number of issues on Github which have been raised that are of a similar theme. I too have tried the suggested netbirdio/dashboard:pr-403 image, but doesn't work.

Please let me know if you would like more info about my setup, config, logs or to support troubleshooting.

[snoopckuu]

Hello everyone,

Happy to confirm that dashboard latest version 2.5.0 released yesterday fixed issue for me!

Thank you!

[git-day]

@snoopckuu , can you please share the version update details?

[snoopckuu]

@snoopckuu , can you please share the version update details?

Sure, i am talking about this release that fixed the issue: https://github.com/netbirdio/dashboard/releases/tag/v2.5.0

[git-day]

@snoopckuu thanks. Is there an easy way to identify what version of the dashboard that has been deployed via docker, given that i can't load the dashboard?

[git-day]

@mlsmaycon can you please help point me in the right direction. How do i troubleshoot this issue? Is there a set of logs to reference and assess where the issue might be? At this stage I have nothing to go by.

[heisbrot]

Hey @git-day ,

you can run docker image inspect netbirdio/dashboard. Under Labels there should be the image version. Latest version as of today is 2.5.0 for the dashboard.

Is your Zitadel instance new? If so you might want to check out the Quick Start Guide. It comes with a one liner to set everything up (including Zitadel, Certificates, etc.). Before running the setup be sure that your domain is pointing to your server.

You can check the logs with docker compose logs dashboard, docker compose logs management