Strange window behavior when running certain display managers

[lee4hmz]

What happened?

Recently, I've been building 86Box from source, and I've noticed that when running windowed and using a GUI inside the guest system, thehost mouse pointer will get stuck on the window decorations and keep me from moving the mouse or using the keyboard until I hit Ctrl-End. I've also noticed that the guest window will sometime stop updating unless I switch away from 86Box to another window and switch back. Full-screen mode does not work at all; it will go full-screen, but the mouse and keyboard will not work. OpenGL rendering is useless, with distorted/overly bold text in text mode and multiple glitches in VGA graphics mode (I tested using the real IBM VGA in the settings).

I did some experimentation, and I found that the window manager doesn't make a lot of difference; I've tried under both GNOME 3's native WM and XFCE's native WM and the same behavior occurs, but only when I'm running GDM 3. SDDM is less problematic, bith in KDE Plasma and GNOME (the issues with the mouse pointer getting stuck and fullscreen not working are gone, but OpenGL rendering is still messed up).

But everything works properly (the mouse pointer, fullscreen and OpenGL rendering) if I use LightDM and XFCE.

I'm wondering just what GDM and SDDM are doing to make 86Box unhappy. It might be a bug in 86Box, since other Qt programs and programs that use OpenGL (like Chrome and Google Earth Pro) are working, but I wanted to open this bug to get some idea as to why the display manager would matter.

Configuration file

[General]
vid_renderer = qt_software
force_43 = 1

[Machine]
machine = ecs386
cpu_family = i386dx
cpu_speed = 40000000
cpu_multi = 1
cpu_use_dynarec = 0
fpu_softfloat = 0
time_sync = local
fpu_type = 387
cpu = 4
mem_size = 8192

[Video]
gfxcard = vga

[Input devices]
mouse_type = msserial

[Sound]
fm_driver = nuked

[Network]
net_01_link = 0
net_02_link = 0
net_03_link = 0
net_04_link = 0

[Storage controllers]
hdc = ide_isa
cassette_mode = load

[Ports (COM & LPT)]
lpt1_device = dot_matrix

[Hard disks]
hdd_01_parameters = 63, 16, 702, 0, ide
hdd_01_ide_channel = 0:0
hdd_01_fn = 340mb.vhd
hdd_01_vhd_blocksize = 4096
hdd_01_speed = ramdisk

[Floppy and CD-ROM drives]
fdd_01_type = 35_2hd
fdd_02_type = 525_2hd

[Other peripherals]
postcard_enabled = 1

Operating system

Debian Linux testing

CPU

AMD Ryzen 5 2400G

86Box version

v4.1 (git commit 38aebdebf7b408543ada5f77c18f9e9dfec673cc)

Build architecture

Linux - x64 (64-bit)

Build type

• [ ] New recompiler
• [ ] Debug build

Download source

I built 86Box myself (please tell us more about your build configuration)

Additional context

Built using the system compiler (gcc (Debian 13.2.0-5) 13.2.0), Qt 5.15.10, and cmake 3.27.7. All dev tools were installed from Debian repositories.