How do you debug/troubleshoot/attach to the CRC VM tty console?

[ofirc]

When downloading the latest version, OpenShift version: 4.18.2 (as of June 18th 2025), from the RedHat Portal, I ran into some cluster responsiveness issues:

[user@crc-host ~]$ crc status
CRC VM:          Running
OpenShift:       Unreachable (v4.18.2)
Disk Usage:      0B of 0B (Inside the CRC VM)
Cache Usage:     28.13GB
Cache Directory: /home/user/.crc/cache
[user@crc-host ~]$ oc get pod
The connection to the server api.crc.testing:6443 was refused - did you specify the right host or port?
[user@crc-host ~]$ ssh crc
ssh: connect to host 127.0.0.1 port 2222: Connection refused
[user@crc-host ~]$

The CRC VM starts fine but then after a while (~1-2h) it stops responding to API calls or SSH.

It's also impossible to see the console of the VM:

[user@crc-host ~]$ sudo virsh console crc
error: internal error: character device serial0 is not using a PTY

[user@crc-host ~]$

And I was wondering whether there is a way to either:

1. Attach to the VM and get an interactive shell, or
2. Viewing the stdout/stderr of the guest VM, or
3. Forwarding the journalctl and/or dmesg logs to the host?

In other words, how does one debug/troubleshoot a stuck/frozen/broken VM?

I looked at the Debugging guide but it assumes that you are able to SSH into the VM, this is obviously not the case above and not necessarily always possible.

It has to be possible somehow (for maintainers and contributors of OpenShift) but the changed is involved. I'm guessing that it requires both attaching a pty to the VM as well as passing a kernel boot argument that forwards the serial to the pty.

Thanks!

[cfergeau]

On linux, you can edit the libvirt configuration of the VM when it’s not running with virsh edit crc, then you can add a serial pty or a vnc graphics device to the config if it’s missing. While ssh is running, you can ssh into the VM and change the password for the core user to a known value.

[ofirc]

@cfergeau I did that but when you try to run: sudo virsh console crc

The stdin isn't passed to the terminal and you don't see anything happening. I think we also need to change the kernel boot arguments to forward/redirect console over serial over pty, but the RHCOS is immutable (AFAIK) so I got stuck there.

Do you happen to have a recipe that works for you and allows you to do that?

[cfergeau]

Quite sure the kernel args can be customized one way or the other, but I don’t know how to do this off hand :-/ I’d go with a VNC console, this lets you access the grub menu, …

[ofirc]

Thanks Christophe! How would I do that with the VNC console and what should be the required changes in the VM XML?

[cfergeau]

Thanks Christophe! How would I do that with the VNC console and what should be the required changes in the VM XML?

https://blog.scottlowe.org/2013/09/10/adjusting-vnc-console-access-via-libvirt-xml/ has some details around this. While editing the XML, look for a preexisting graphics node and replace it with <graphics type='vnc' port='-1' autoport='yes'/>

[ofirc]

@cfergeau thanks! I'm wondering what's the easiest way to see the console. My setup includes: MacOS with VSCode -> GCE instance (host VM) -> CRC I think VSCode would handle local port forwarding from the desktop (MacOS) to the remote machine (GCE instance) and I'd need to point the VNC client to the local port on the Mac.

A lot of hops along the way, it's doable but seems quite intricate and involved.

[cfergeau]

I don’t know about vscode, does your GCE instance have a public IP? you could set the VNC server to listen on 0.0.0.0 (with a password), and connect to this. If it does not have a public IP, you’d need to do port forwarding somehow, or put your local machine and the GCP instance on the same network using tailscale or similar.