cockpit-project
Blank Page if /etc/cockpit/machines.d has unreadable files
Explain what happens
- Cockpit
:9090 can login and dashboard is accessible to manage the local machine. - I add another host to manage it on this installation of cockpit.
- Upon logout and logging back in, it's showing a blank page.
Version of Cockpit
323.1-1.el9_5
Where is the problem in Cockpit?
Overview
Server operating system
Red Hat Enterprise Linux
Server operating system version
9.5
What browsers are you using?
Firefox, Chrome, Edge
System log
journalctl did not capture anything critical, even after my attempt of enabling the debug mode for cockpit
-- Boot 01bdf8a78272433e8e458e511f960fb6 --
Nov 22 10:31:55 private01.cvad.unt.edu systemd[1]: Starting Cockpit Web Service...
Nov 22 10:31:55 private01.cvad.unt.edu systemd[1]: Started Cockpit Web Service.
Content of /etc/systemd/system/[email protected]
[Unit]
Description=Cockpit Web Service https instance %I
Documentation=man:cockpit-ws(8)
BindsTo=cockpit.service
[Service]
Environment=G_MESSAGES_DEBUG=cockpit-ws,cockpit-bridge
Slice=system-cockpithttps.slice
ExecStart=/usr/libexec/cockpit-ws --for-tls-proxy --port=0
User=cockpit-wsinstance
Group=cockpit-wsinstance
https://github.com/user-attachments/assets/51beb7ca-6ec8-4eb5-92fa-37d541b8490a
I came to discover that this issue is caused by a custom build of RHEL 9(.5) with a CIS-Server Level 1 benchmark policy. I tested a RHEL 9.5 with no policy and it works. Unfortunately, I can't trace anything useful with the Cockpit debug logs.
I don't know what a "CIS-Level 1 benchmark policy" is -- can you roughly describe what that is? I suppose https://www.cisecurity.org/cis-benchmarks but that is very abstract. A kernel change, a browser plugin, some security restrictions, etc?
In the video, what's the difference between the left and right browser? they seem roughly equivalent, and at least talk to the same host IPs/names, but the right one is called "ScreenConnect" - some kind of remote desktop?
I think the bug happens at 2:27, right? The journal spits out a lot of TLS errors, and one more when you login. That may just be browser dependant, but it's worth taking a look at the browser console (Ctrl+Shift+J) -- open it on the login page, then log in, and see what happens. Can you please copy the messages here?
No response in two months, closing. I'm happy to reopen when you send the requested information. Thank you!
I'm seeing the same thing with the stock configuration in Almalinux 9.
Steps:
- enabled and start cockpit
- Accessed in firefox at localhost:9090
- Added another server to the console
- Refreshed the browser
- Get blank screen
- Logged out and back in again, still the same
- A colleague has experienced the same behavior on his system.
- All system packages were updated about 3 weeks ago.
- We weren't using cockpit at all prior to this test
@martinpitt I think you can reopen this one. I am also seeing this on fresh install of RHEL 9.5 with installed security profile CIS RHEL 9 Benchmark for Level 1 - Server and cockpit 323.1-1.el9_5. When you add new server to the console the issue start to happen. Tested with Firefox 135 and Chromium 133.0.6943.98.
The only suspicious thing in the journal can be:
cockpit-tls[42955]: cockpit-tls: gnutls_handshake failed: A TLS fatal alert has been received.
cockpit-session[43353]: pam_ssh_add: Failed adding some keys
The first entry is probably due to self-signed certificate (this is also visible when cockpit was working correctly). So maybe it is with ssh key ? Maybe default one from cockpit is in conflict with ssh security policy for RHEL 9 ? (I don't know this is only assumption)
$ sudo update-crypto-policies --show
DEFAULT:NO-SHA1:DISABLE-INSECURE-INSIGHTS
About security profile - you can select one during RHEL instalation or introduce manually later - see attached screenshots:
Anyway when you delete /etc/cockpit/machines.d/99-webui.json (so you remove host which was added via cockpit gui) everything is back to normal and no more "blank page" is visible after you logon to cockpit.
With any luck this is a duplicate of https://issues.redhat.com/browse/RHEL-78645 aka. #21606.
But /etc/cockpit/machines.d/99-webui.json has absolutely nothing at all to do with TLS and crypto policies. Most probably/hopefully this is just a red herring, and the real change was something else?
Well it could be, in RHEL9 with DEFAULT crypto policies, some of crypto algorithms are removed and also for RSA there is RequiredRSASize 2048. I will do the test with LEGACY crypto policy and let you know.
Regarding /etc/cockpit/machines.d/99-webui.json, yes I know, this is where those additional hosts to manage are defined. So deleting this simply reverted operation of adding additional host to manage via gui.
I recently successfully reproduced the issue like this:
# chmod 600 99-webui.json
This makes all non-root user go blank page but root login works [root is by default disallowed so I guess original poster faced the issue with non root only]
Strace of cockpit-bridge discovered this issue:
2380296 10:09:59.597542 openat(AT_FDCWD</>, "/etc/cockpit/machines.d/99-webui.json", O_RDONLY|O_CLOEXEC <unfinished ...>
2380296 10:09:59.597680 <... openat resumed>) = -1 EACCES (Permission denied) <0.000109>
2380296 10:09:59.597884 sendmsg(10<UNIX-STREAM:[8031467->8031466]>, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\3\1\1J\0\0\0\35\0\0\0?\0\0\0\5\1u\0\33\0\0\0\4\1s\0'\0\0\0org.freedesktop.DBus.Error.AccessDenied\0\10\1g\0\1s\0\0", iov_len=80}, {iov_base="E\0\0\0[Errno 13] Permission denied: '/etc/cockpit/machines.d/99-webui.json'\0", iov_len=74}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL <unfinished ...>
Good chance to either improve cockpit logging to make it log basic permission denied issues to the logs or if at least skip the file if it's not accessible and proceed serving further files.
In Network tab of browser you will notice only few files/js/favicon is loading and nothing else.
Thanks @ShreyasMahangade for figuring this out! This reproduces on main. Oh dear, let's fix this.