libsdl-org
SDL_WINDOW_UTILITY does absolutely nothing on MacOS
Creating a window on MacOS with or without the SDL_WINDOW_UTILITY flag results in the same window being created. It still appears in the app switcher and the dock.
SDL_WINDOW_TOOLTIP and SDL_WINDOW_POPUP_MENU on MacOS result in the window not appearing. It also appears in the app switcher and dock.
SDL_WINDOW_NOT_FOCUSABLE makes the resulting SDL3 window appear unfocussed (the window border is greyed out), but it still definitely takes focus.
SDL_WINDOW_TOOLTIP and SDL_WINDOW_POPUP_MENU on MacOS result in the window not appearing. It also appears in the app switcher and dock.
Can you provide a repro case for this behavior? I'm not seeing any odd behavior with testpopup.
On macOS, it's the application that shows up on the dock and task switcher, so SDL_WINDOW_UTILITY has no effect.
These flags are distinct window types, you can't set more than one of these at a time:
(SDL_WINDOW_UTILITY | SDL_WINDOW_TOOLTIP | SDL_WINDOW_POPUP_MENU | SDL_WINDOW_MODAL)
On macOS SDL_WINDOW_NOT_FOCUSABLE prevents the window from becoming key, e.g. having title bar highlighting indicating input focus. It also prevents the window from getting keyboard focus.
I changed it so on macOS we don't get keyboard events if no window has focus, which matches other desktop platform behavior, in https://github.com/libsdl-org/SDL/commit/51fa076fdc56c3d9da92bf2f25bd78a23b80362e.
I believe this addresses all of your concerns. Please feel free to reopen if there's a bug in my changes or you need further clarification.