Mouse & Debugger under Linux + Make SDL_CaptureMouse optional

[pfirsich]

Version/Branch of Dear ImGui:

Version: 1.80 WIP Branch: master (bff18369224193bb33402f6a6f757d926afb37ff)

Back-end/Renderer/Compiler/OS

Back-ends: imgui_impl_sdl.cpp Operating System: Ubuntu 20.04 (GNOME Shell)

My Issue/Question:

I am using ImGui to render in-game UI in an FPS and SDL_CaptureMouse will interact weirdly with SDL_SetRelativeMouseMode (required/recommended for FPS style mouse capturing). Clicking, which will result in SDL_CaptureMouse(SDL_TRUE) being executed once, will put the mouse cursor in a weird state, where it remains invisible on the application's window, but becomes visible on surrounding windows (which then become clickable), so you can lose focus. Even if I do SDL_SetRelativeMouseMode(SDL_TRUE) every frame after ImGui rendering the mouse remains in that weird state until I tab out of the application and back in again. Simply clicking on another window and clicking back will not fix it. If I replace the call with SDL_CaptureMouse(SDL_FALSE), the problem does not occur, so the issue is caused by calling calling it with SDL_TRUE. This even happens in fullscreen mode with multiple monitors. I am sure this is also a GNOME bug or an SDL2 bug or I don't even know, but I think it would be good if the SDL_CaptureMouse(SDL_TRUE) call in ImGui_ImplSDL2_UpdateMousePosAndButtons could be prevented with some ImGuiIO flag or something. It seems to have been added for #1559, so it definitely has its place, but it would be nice to toggle that behaviour, so I don't have to use my own imgui_impl_sdl.cpp-file just to remove that line.

Thanks for all your work. ImGui is truly amazing!

[vicenteferrari]

I stumbled upon this exact problem, figured it out as well and came to post an issue.

Indeed I also fixed it by commenting out the SDL_CaptureMouse in ImGui_ImplSDL2_UpdateMousePosAndButtons. I also think it's window manager independent and related to X11.

[sturnclaw]

@pfirsich while the codebase I work on also does not use the SDL_CaptureMouse path due to window positioning issues on certain DEs, you should consider calling SDL_SetWindowGrab() as well as SDL_SetRelativeMouseMode() or SDL_CaptureMouse(). This will constrain the mouse within the window coordinates, preventing it from leaving the window and accidentally clicking outside. When this is enabled while the mouse is dragging/hidden, I've never had mouse-leakage issues with SDL_CaptureMouse().

[audetto]

I support the idea to review the usage of SDL_CaptureMouse.

I find that it makes debugging extremely hard when a breakpoint is triggered by a mouse click. Almost all the times, and this has been observed on Ubuntu 20.10 (GNOME) and Pi4.

If the mouse is still in grabbed state when the breakpoint is hit, the debugger (and all other desktop apps) will not received mouse clicks, making their usage extremely complicated.

And definitely, I'd like to be able to switch it off without having to carry a modified version of ImGui.

[ocornut]

Hello,

Came here from #3956 which also suggest an issue with debugger breaking during capture and losing mouse inputs in other applications.

(1) For the specific issue of conflicting with SDL_SetRelativeMouseMode(), would doing this work?

if (SDL_GetRelativeMouseMode() == SDL_FALSE)
{
    bool any_mouse_button_down = ImGui::IsAnyMouseDown();
    SDL_CaptureMouse(any_mouse_button_down ? SDL_TRUE : SDL_FALSE);
}

(2) For the issue that debug breaking while capturing appears to conflict, mentioned in #3956 and https://github.com/ocornut/imgui/issues/3835#issuecomment-803447395, I think we can consider a workaround in the form of an opt-out, but I would wouldn't like this workaround to be a lazy solution for (1). A better workaround may be to configure the app to react to debugging signal (SIG_TRAP????) and drop capture? I also firmly believe this is not a Dear ImGui backend problem, rather than a SDL+X11 problem and should be reported to SDL, but we can do our best to provide a short-term workaround if needed.

What SDL_CaptureMouse() does:

• Under Windows at least: without SDL_CaptureMouse() when clicking on dear imgui widget and dragging outside the platform window, Win32 would react to hovering platform window borders and force e.g. resize cursor. Can you confirm the behavior without capture on OSX and X11? Would any other windows/app react to hovering contents while click+dragging off our app window?
• It allows to access mouse coordinates while dragging outside and without using SDL_GetGlobalMouseState() which is not always available. If we click a dear imgui widget and drag outside the host platform window and have neither capture neither SDL_GetGlobalMouseState(), mouse coordinates become stuck at the edge of windows which is very detrimental to many interactions. (For reference this was first added in #1559 with commit f3e510a9bf508337244433f8c1dcc6c120dcc990 then reworked)

[audetto]

Mouse & Debugger: https://discourse.libsdl.org/t/debugging-with-sdl-capturemouse/31014 I think this is considered a feature, and I suspect the idea it not to abuse SDL_CaptureMouse.

But, good thing is that the suggested method works.

I got it working with

def release_mouse (event):
    gdb.write("GDB/SDL2: Releasing mouse\n")
    gdb.execute("call SDL_CaptureMouse(0)")

Guess I need to be careful if SDL is not loaded, but it seems to provide a solution.

Might be a good one to add to the FAQs.

[rokups]

Actual solution for maintaining input on X11 is this:

setxkbmap -option grab:break_actions
xdotool key XF86Ungrab

[audetto]

I have just realised that Qt has the same problem and their solution is via a command line switch.

https://doc.qt.io/qt-5/debug.html#command-line-options-recognized-by-qt

They are able to detect gdb and disable the grabbing.

Which I think a good option as it avoids the whole setup required by the solution I linked above.

[audetto]

I've come here again, after a system upgrade and change X11 / Wayland.

Even if I have this gdb hook:

gdb.execute("call SDL_CaptureMouse(0)")

it does not work any longer. I was wondering if I could propose a PR with a new (optional) MACRO: IMGUI_DISABLE_SDL_MOUSE_GRAB which is a much easier solution that trying to workaround SDL / gdb / X / Wayland etc....

[ocornut]

At minimum it shouldn’t be a compile time option but a function to enable/disable the capture, so you can selectively disable them when a debugger is attached.

Note that it will likely break other things. So really you would be better off investigating another solution.

[rokups]

Dear ImGui behaves correctly here. You should blame X11 design for what it is. The correct way around this problem is manually breaking mouse capture:

# Enable ungrabbing at will. Needs to be called once at boot.
setxkbmap -option grab:break_actions
# Do actual ungrabg
xdotool key XF86Ungrab

Bind it to a hotkey or something for more convenience.

[audetto]

It takes 2 to tango. If I force SDL to use Wayland, then SDL_CaptureMouse is not implemented and it returns -1 always.

This is either good for everybody (it just works) or become an issue for ImGui which will need to find an alternative solution.

I can't really understand it, but this maybe is a good sign: https://github.com/libsdl-org/SDL/commit/f600364b8a575e03f140ab18eeabaeffb8f0b392

Next SDL (2.27) will likely make Wayland the default when available.

[ocornut]

I don't think you should expose a backend agnostic feature in IO if it's not supported by all backends.

PR a SDL backend function.

But if your issue is specific to Wayland then perhaps the SDL backend can handle this specificity itself.

It's hard to trust a PR if you don't put the effort of understanding why we normally need the capture on other systems, in order to decide what is actually the best design/solution for e.g. Wayland.

[audetto]

Fine, scrap that, it was not good. The new one is just for SDL2, but I can add SDL3 if it looks good.

I admit I don't not fully understand what happens, but at the same time I know I cannot debug unless I disable capture. In Ubuntu 22:10 I was using X11 and this trick was enough

#!/usr/bin/python

def release_mouse (event):
    gdb.write("GDB/SDL2: Releasing mouse\n")
    gdb.execute("call SDL_CaptureMouse(0)")

gdb.events.stop.connect(release_mouse)
gdb.write("GDB/SDL2: installed release mouse for SDL2\n")

On Wayland, SDL still prefers X11, and I have no idea about the interaction of SDL, ImGui, XWayland and Wayland, so I can only propose a solution that I understand.

Qt offers the exact same solution: https://doc.qt.io/qt-6/debug.html#command-line-options-recognized-by-qt so, without going crazy, I have simply copied the same approach.

EDIT: and, it is very possible that disabling capture has side effects, but as long as I can debug the rest 90%, I can live with it.

[lailoken]

An easy way for me to work around this without needing to run apps etc (for all the developers) was to add the following change to my SDL backend.

I replaced:

    SDL_CaptureMouse((bd->MouseButtonsDown != 0) ? SDL_TRUE : SDL_FALSE);

with:

#ifdef _DEBUG
    // This breaks debugging mouse actions (click) under Linux / Gnome / X11
    // only capture mouse while dragging (left mouse and movement)
    if (ImGui::IsMouseDragging(ImGuiMouseButton_Left))
        SDL_CaptureMouse(SDL_TRUE);
    else
        SDL_CaptureMouse(SDL_FALSE);
#else
    SDL_CaptureMouse((bd->MouseButtonsDown != 0) ? SDL_TRUE : SDL_FALSE);
#endif

The ImGui::IsMouseDragging check is needed to make window dragging less annoying.

[ocornut]

My intuition is that we could largely mitigate the issue by using:

bool want_capture = false;
for (int button_n = 0; button_n < ImGuiMouseButton_COUNT && !want_capture; button_n++)
    if (ImGui::IsMouseDragging(button_n, 1.0f))
        want_capture = true;
SDL_CaptureMouse(want_capture);

AKA only start capturing after mouse has started moving with a button down. Could affected people try this?

[audetto]

I will try, but at the same time this issue is less and less important since Wayland does not offer the ability to capture the mouse.

Compare

https://github.com/libsdl-org/SDL/blob/f0f593f04b574f5e0657970b2b679912d4f07444/src/video/wayland/SDL_waylandmouse.c#L976-L983

to

https://github.com/libsdl-org/SDL/blob/f0f593f04b574f5e0657970b2b679912d4f07444/src/video/x11/SDL_x11mouse.c#L499

Forgot to say, in Wayland, it returns -1, "That operation is not supported".

[ocornut]

I have pushed aa83d5d since I believe it would be harmless and mitigate the issue.

[ocornut]

So under Wayland, how does it works when you e.g. click to start dragging an imgui window, and move the cursor outside the boundaries of the platform window? Does the imgui window gets "stuck" or does it keeps moving (while being clamped by imgui) ?

How does SDL provide a reliable mouse position during that time?

[rokups]

I am developing on wayland currently. Debugger definitely does not lock input like X11 does. I think window merely gets messages when cursor moves over the window. Also, if cursor is moved over a window that is paused in debugger it causes a soft crash. Linux, what can i say...

[rokups]

I verified what happens when window is not processing events: queue is filled up and if process resumes execution in timely manner then all queued events are processed by the application and everything continues properly. Unless event queue overflows that is, then there is that soft crash where window disappears, but process remains running. tested with SDL2.