GNOME desktop freezing on VMware Workstation Pro for Windows host and Fedora guest (likely due to 3D acceleration)

[hermidalc]

Hello VMware Development Team,

I have a new Windows laptop running latest Windows 10.0.15063 and VMware Workstation 12.5.7 build-5813279. It uses hybrid graphics between integrated Intel and Nvidia GPUs using Nvidia Optimus with default settings on a 4K display. VMware runs on the Intel HD 630 GPU (verified by Nvidia GPU activity monitor never showing VMware process). In the BIOS I've set for the Intel GPU to get the max amount dedicated video memory possible (512 MB). I created a new guest running the latest Fedora 25, with 2 GB max guest memory for graphics. open-vm-tools and open-vm-tools-desktop (10.1.5-4.fc25) installed automatically and successfully.

The Gnome desktop will just randomly freeze and I cannot do anything or get control of it without doing a hard reboot of the VM. After a lot of troubleshooting to figure out what is causing the freezing it appears to be due to the 3D acceleration and only in full screen mode. The freezing will occur in both XOrg and Wayland desktop sessions. If I turn off 3D acceleration or not have the VM in full screen (4K) then it doesn't freeze, but this is very inconvenient as Gnome doesn't run well without 3D acceleration.

$ glxinfo | grep OpenGL OpenGL vendor string: VMware, Inc. OpenGL renderer string: Gallium 0.4 on SVGA3D; build: RELEASE; LLVM; OpenGL core profile version string: 3.3 (Core Profile) Mesa 17.0.5 OpenGL core profile shading language version string: 3.30 OpenGL core profile context flags: (none) OpenGL core profile profile mask: core profile OpenGL core profile extensions: OpenGL version string: 3.0 Mesa 17.0.5 OpenGL shading language version string: 1.30 OpenGL context flags: (none) OpenGL extensions: OpenGL ES profile version string: OpenGL ES 3.0 Mesa 17.0.5 OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.00 OpenGL ES profile extensions:

[MaZZly]

Similar build and same problem here: Windows 10 Pro, 1703, Build: 15063.483 Fedora 26.

Running full-screen (4k monitor) seems to bring freeze randomly. Sometimes I can work for an hour in the VM, but most times it freezes after several minutes.

When running windowed mode it doesn't seem to happen.

I haven't tried disabling the 3D acceleration.

[MaZZly]

After testing with 3D acceleration off yesterday I can confirm that no crashes happen during this time. Painfully slow to work with though..

[hermidalc]

I can confirm the same, I've tested both Fedora 25 and 26 identical VM guest installs, using Xorg (not Wayland), and tested using open-vm-tools or proprietary VMWare Tools. GNOME desktops freeze randomly after a few minutes use unless I turn off VMware 3D acceleration which honestly isn't a solution GNOME needs 3D acceleration to be usable.

I also tested Fedora 24 with the same setup and it works and does not freeze.

[serpixel]

This also occurs on Ubuntu 17.04. A workaround is to modify /usr/share/xsessions/gnome.desktop and set Exec=env LIBGL_ALWAYS_SOFTWARE=1 gnome-session --session=gnome

[hermidalc]

Thanks for the additional workaround @josepmc, may I ask though is this any different or better performing than my workaround where you disable VMware 3D hardware acceleration for the VM? Turning off hardware acceleration in GNOME makes it almost unusable.

I hope the open-vm-tools team can figure out why, in Fedora 25 and newer using XOrg (not Wayland) and I assume Ubuntu 17.04 and newer, there is this desktop freezing issue that makes it impossible to have these guest OSs in VMware.

[serpixel]

This does disable it system wide. However you can enable it for other applications by setting the environment variable to 0 for them.

This makes the system go smoother, but it's by far not the desired solution.

On 26 Sep 2017, at 01:00, Leandro Hermida <[email protected]mailto:[email protected]> wrote:

Thanks for the additional workaround @josepmchttps://github.com/josepmc, may I ask though is this any different or better performing than my workaround where you disable VMware 3D hardware acceleration for the VM? Turning off hardware acceleration in GNOME makes it almost unusable.

I hope the open-vm-tools team can figure out why, in Fedora 25 and newer using XOrg (not Wayland) and I assume Ubuntu 17.04 and newer, there is this desktop freezing issue that makes it impossible to have the guest OSs in VMware.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/vmware/open-vm-tools/issues/176#issuecomment-332037829, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AE2i2k8BptxtEs-e-EGJactKcuZVpVJ9ks5smDCDgaJpZM4OMTxy.

[oliverkurth]

Thanks for reporting. I believe this is an issue with Workstation (I may be wrong), and filed an internal bug.

[hermidalc]

I can also confirm that on the latest VMware Workstation 14 Pro (14.0.0 build-6661328) this major problem still exists. I created a brand new Fedora 26 guest OS using my exact same procedure mentioned above and after using the GNOME desktop for a few minutes in 4k resolution everything freezes.

[thomashvmw]

Hi! I tried to reproduce with an older hybrid graphics laptop but never saw the problem. Not running 4K, though.

Could anyone seeing this problem try one of the following suggestions to help pinpoint the problem:

1. In the laptop bios setup, under Video, turn off Nvidia Optimus and then relaunch win 10 and retry?
2. In the Nvidia control settings, select Nvidia graphics as the system-wide default instead of auto?

Thomas

[hermidalc]

Hi - sorry I should've mentioned this, when troubleshooting I did turn off hybrid graphics in BIOS and play around with Nvidia control settings and it doesn't matter the GNOME desktop will still freeze.

[thomashvmw]

Anyone that could post a vmware.log from a failing VM?

is it possible to ssh to a failing VM and get the output of "dmesg"?

[serpixel]

It does happen under Intel graphics machines too.

On 02 Oct 2017, at 16:06, thomashvmw <[email protected]mailto:[email protected]> wrote:

Anyone that could post a vmware.log from a failing VM?

is it possible to ssh to a failing VM and get the output of "dmesg"?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/vmware/open-vm-tools/issues/176#issuecomment-333562629, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AE2i2vRIMZrdIxLRl4zWH60vJUAu_Z4wks5soPvVgaJpZM4OMTxy.

[hermidalc]

@thomashvmw I've had a support ticket open with VMware regarding this issue and they had me generate all this extra logging of the failing VM, the logs don't seem to show anything pointing to the source of the problem. They had me make this /etc/vmware-tools/tools.conf file:

log = true

# Enable tools service logging to a file
vmtoolsd.level = debug
vmtoolsd.handler = file
vmtoolsd.data = /tmp/vmtoold.log

# Enable 'vmsvc' service logging to a file
vmsvc.level = debug
vmsvc.handler = file
vmsvc.data = /tmp/vmsvc.log

# Enable VMwareResolutinSet.exe logging to a file
vmresset.level = debug
vmresset.handler = file
vmresset.data = /tmp/vmresset.log

# Enable new "vmusr" service logging to a file
vmusr.level = debug
vmusr.handler = file
vmusr.data = /tmp/vmusr.${USER}.log

# Enable the 'vmvss' snapshot service logging to a file
vmvss.level = debug
vmvss.handler = file
vmvss.data = /tmp/vmvss.log

And give them the logs. About dmesg, I can generate a dmesg output, apologies since I don't know much about it, should I generate it after the guest OS GNOME locks up or anytime?

[hermidalc]

Ok I started up the new Fedora 26 guest OS I made and after a couple minutes as usual GNOME froze, I sshed into it and ran dmesg, here's the output

dmesg.log

Seems like a clue here, line 1715:

[   25.181322] vmtoolsd[2141]: segfault at 14c0 ip 00000000000014c0 sp 00007ffe6c707218 error 14 in vmtoolsd[55b98b6cd000+ac000]

[thomashvmw]

Hi. No, the vmtoolsd error is unrelated. While it's bad, it shouldn't cause a freeze. It would also be helpful if you could find and post the vmware.log file which is located in the vm's folder on the host.

Thanks, Thomas

[hermidalc]

I restarted the VM and ran dmesg again while GNOME is working and unfortunately that message is the last line in the dmesg log after startup:

[   16.373174] vmtoolsd[2152]: segfault at 14c0 ip 00000000000014c0 sp 00007ffde077e368 error 14 in vmtoolsd[5615b79b9000+ac000]

So not sure really if it's a clue anymore

[hermidalc]

Here you go, used the GNOME desktop on restarted VM and let it freeze again

vmware.log

[mumblyOMOD]

I have the same issue. My host is a Windows 10 desktop (ver 10.0.15063) with both a AMD HD 6950 and and AMD RX580 GPU. My guest system is a fully updated Manjaro. I've been running without 3D acceleration for over a year. I'm not sure if anything's changed, but last time I had 3D acceleration enabled the entire Gnome UI would sporadically crash, but I could still get into a terminal using keyboard shortcuts (eg Ctl+Alt+F2). I'm happy to provide any of my configuration files or logs if they'd be of any help.

[abecher22]

Same problem here. Win10, Workstation 14 Pro, RHEL7 guest. Also, turning off 3D acceleration causes multi-monitor mode to display black screens.

[hermidalc]

@abecher22 I think yours is a different issue. Here we are describing a major issue where GNOME, on guest Linux OSes Fedora 25 and Ubuntu 17.04 and newer, will always freeze if you have VMware 3D acceleration turned on.

By default 3D acceleration should always be on. Most modern Linux graphical desktops need it to function properly that's why it's a major issue for me and others because you effectively cannot use VMware Workstation with any late guest OS.

My workaround is to simply use Fedora 24 as guest OS but eventually that will be a problem because for one reason or another I will need to upgrade. Turning off 3D acceleration is not a workaround the desktop is really unusable if you do that.

I wholeheartedly believe that if the VMware Workstation development team or the open-vm-tools development team took a Windows 10 computer with VMWare Workstation 12.5.x or 14.x and a 4K monitor and installed Fedora 26 guest OS with 3D acceleration turned on they will see this issue within 10-15 minutes of use. The GNOME desktop will always eventually completely freeze.

The reason I ask to use a 4K display is because it seems like it might be that this configuration is causing VMWare Workstation to have problems. I haven't yet seen anyone say this exact issue happens on HD? @mumblyOMOD is yours 4K?

[abecher22]

@hermidalc I am also experiencing the GNOME freeze. I am using 2 4k monitors. I was just pointing out that when I disable 3D acceleration (which avoids the freeze), I cannot use my dual monitors. I should have made that clear.

[mumblyOMOD]

@hermidalc I generally use a dual monitor setup with both screens at 1920x1200. I think the issue happened to me when only using one monitor, but it's been so long since I've had 3D acceleration enabled that I'd need to test it again to be completely sure.

@abecher22 It might not be of any help to you, but with 3D acceleration disabled I am able to use two of my 1920x1200 monitors without any issues.

[abecher22]

@mumblyOMOD I cannot use my dual 4k monitors with 3D acceleration turned off. GNOME simply "blacks out". I can use one of the monitors, but not both. With 3D acceleration enabled I can use both 4k monitors but then I have to deal with periodic freezing.

[hermidalc]

Thanks for the follow up and clarification @abecher22 and @mumblyOMOD. More info and data points for the VMware Workstation team to show this is a real problem.

I'm going to hazard a guess and say that this issue has something to do with a flaw in VMware Workstation and its 3D acceleration using late version Linux guest OSes on single 4K and dual HD monitors. Could it be some flaw there isn't enough memory somewhere? My guess is that a single 4K or dual HD displays will require much more memory for 3D acceleration, etc. We are all setting our video memory to the max 2 GB in the settings.

[thomashvmw]

Hi all. Unfortunately we have not been able to reproduce this in-house, so we do need some more help to pinpoint which component is at fault. If you have the possibilty to run any (0-2) of these tests, it would be helpful.

0. If someone has a VM that frequently locks up (Let's call it "Locking"), a) Locate the "Locking" VM folder on the host. b) Edit the "Locking.vmx" file by adding the following two lines at the end: mks.enableSoftwareRenderer=1 mks.enableDX11Renderer=0 This will still have the VM think that 3D is available, but will direct all VM 3D operations to Workstation's own software 3D renderer rather than to the host's DX11 subsystem, so things will be a bit slow... c) Check whether the VM still locks up, and regardless of the result, please post the "vmware.log" file present after the run in the "Locking" VM folder on the host.

1. If someone has a VM the frequently locks up, and finds an error in the "dmesg" guest log or the Xorg log (typically /var/log/Xorg.0.log) in the guest, that would be of great help.

2. If someone has a VM that frequently locks up and can try the exact same VM using a Linux Player or Workstation setup to determine whether there is a lockup also on linux hosts, that would be of great help.

Thanks, Thomas

[thomashvmw]

And another simple test that would also be useful if you're running on a Nvidia system (Optimus disabled if multi-GPUs): 3. Exactly the same as 0., but replace mks.enableSoftwareRenderer=1 with mks.enableGLRenderer=1 This will direct the VM's 3D operations to the host's Nvidia OpenGL driver.

Thanks, Thomas

[mumblyOMOD]

@thomashvmw I'll try turning on 3D acceleration today and will provide logs for #1 sometime in the next 8 hours or so. After it exhibits the Locking behavior I'll tweak my vmx file according to #0 and will provide an update once I've tested it for a good 8 hours or so (enough time that I know my machine should have exhibited the Locking behavior at least once).

[hermidalc]

@thomashvmw sorry to ask for clarification on your earlier comment, were you able to reproduce the issue in-house or not? Sorry it's unclear from "Unfortunately we have been able to reproduce this in-house"

[hermidalc]

I followed instructions for test 0 above on my test Fedora 26 guest OS. It locked up within 15 seconds after booting, doing what I typically do to test it. I always open up a couple desktop windows, i.e. a terminal and firefox, and then test GNOMEs desktop functionality like alternating between Activities view and the desktop, and switching between desktop and windows. All of these test GNOME's basic 3D capabilities.

dmesg.log vmware.log

[thomashvmw]

@hermidalc, No we have not been able to reproduce internally, unfortunately. I've updated the previous comment