Remove Window Shape API

[icculus]

We talked briefly about having an option to make a window's alpha channel mask out the actual window, which some platforms can do. With or without this, I think we should drop the existing shape API, which mostly doesn't work.

[slouken]

I think we should officially implement the window's alpha channel mask before removing the shaped window API. This is an API I happen to know gets used occasionally, and we should have a migration path here.

[1bsyl]

Looking at the window shape api, it seems working with some bug. For instance RecreateWindow is probably not recreating the shaper underneath.

Not sure how we should improve the API with window's alpha channel mask ?

Maybe moving out the shape public interface to internals, and only expose a simpler function:

int SDL_SetWindowShapeSurface(SDL_Window *window, SDL_Surface *shape)
{
    SDL_WindowShapeMode mode;

    if (window == NULL) {
        return SDL_InvalidParamError("window");
    }    
    if (shape == NULL) {
        return SDL_InvalidParamError("shape");
    }

    if (SDL_GetVideoDevice()->shape_driver.CreateShaper == NULL) {
        return SDL_Unsupported();
    }    

    if (window->shaper == NULL) {
        window->shaper = SDL_GetVideoDevice()->shape_driver.CreateShaper(window);
        if (window->shaper) {
            window->shaper->userx = window->x;
            window->shaper->usery = window->y;
            window->shaper->mode.mode = ShapeModeDefault;
            window->shaper->mode.parameters.binarizationCutoff = 1; 
            window->shaper->hasshape = SDL_FALSE;
        } else {
            return SDL_SetError("cannot create shaper");
        }    
    }    

    if (SDL_ISPIXELFORMAT_ALPHA(shape->format->format)) {
        mode.mode = ShapeModeBinarizeAlpha;
        mode.parameters.binarizationCutoff = 255;
    } else {
        SDL_Color black = { 0, 0, 0, 0xff };
        mode.mode = ShapeModeColorKey;
        mode.parameters.colorKey = black;
    }    

    return SDL_SetWindowShape(window, shape, &mode);
}

[slime73]

Maybe moving out the shape public interface to internals, and only expose a simpler function

I think the idea is to remove all of the window shape APIs (internally and externally), and add new functionality similar to GLFW's GLFW_TRANSPARENT_FRAMEBUFFER window flag to allow apps to render any shape they want just by rendering to the backbuffer.

I believe all major desktop windowing APIs support that concept, so it's just a matter of implementing it for each platform.

[slouken]

Maybe moving out the shape public interface to internals, and only expose a simpler function

I think the idea is to remove all of the window shape APIs (internally and externally), and add new functionality similar to GLFW's GLFW_TRANSPARENT_FRAMEBUFFER window flag to allow apps to render any shape they want just by rendering to the backbuffer.

I believe all major desktop windowing APIs support that concept, so it's just a matter of implementing it for each platform.

Yes, this is the goal. Does anyone want to take a crack at this?

[1bsyl]

@slime73 thanks for the feedback!

I can start a PR for this .. it will be very likely incomplete.

how should we expose to have a transparent framebuffer. Hint ? Window Flags ? SDL_GL_Attribute ? other ?

[1bsyl]

Remove Shaped Window here: PR https://github.com/libsdl-org/SDL/pull/7316 And I will try on a different PR to add Transparent stuff.

I've tried glfw first to see how it worked.

x11 / GLX: ok x11 / egl: doesn't worked

wayland / glx: ok wayland / egl: doesn't worked

And trying with SDL, and so far I can get: x11 / GLX: ok x11 / egl: ok

wayland/glx: we don't have wayland with glx here! wayland/egl: ok

[slime73]

how should we expose to have a transparent framebuffer. Hint ? Window Flags ? SDL_GL_Attribute ? other ?

A window flag makes sense to me I think - I believe it should work with Metal / Vulkan / Direct3D surfaces, so it's not an OpenGL-specific thing.

[1bsyl]

here's some PR https://github.com/libsdl-org/SDL/pull/7319

[icculus]

That PR got merged (and there is a separate open issue in #7460 about adding Windows support).

Do we want to kill the Shape API now? There's a PR for that in #7316.

[slouken]

No, not yet. I need to review how Steam uses it and see if the transparent window flag can be used instead.

[serpilliere]

Hi!

Is the decision of removing the shape API definitive? @icculus : when you said 'which mostly doesn't work.', do you mean it doesn't do the job it's made for, for too many video drivers ? or from a performance perspective?

If I read correctly, it seems you are moving from the shape API to supporting full alpha channels to render with transparency.

If yes, I have a question on this: Let say we have an application which constantly changes its window shape. Today, some native drivers (windows gui, x11, cocoa, ...) support shape changing by sending a bunch of rectangles/polygons to define the final shape, which is quite fast. If I read correctly, SDL already forces shape inputs as bitmap so for some of those drivers, SDL does the job of transforming those bitmaps to bunch of rectangles (with SDL_CalculateShapeTree if I am correct), even if the driver supports "rectangles shape API".

If everything is moved to alpha canal, applications will have to generate big buffers to send to the driver, which, for 4k screen for example, may be quite cpu intensive. Moreover, even if the video driver doesn't support "rectangle shape API description", it may at the end generate the bitmap buffer composed by those rectangles internally.

(sorry if it's not the place to talk about this)

[icculus]

This is definitely the place to talk about this.

So the shape API code hasn't been maintained in years, and it definitely didn't work on some major platform (macOS, I think). I don't know where it stands on X11, I don't know if it's even implemented for Wayland.

If we can clean it up and make it work on all reasonable platforms, I'm not against keeping it. But I don't know if that's possible.

It does seem like platforms are moving towards having surfaces with alpha channels that will blend with the desktop, because they generally have these, and a compositor, all cooking on the GPU, whether you use them directly or not. But I'm happy to be wrong.

[serpilliere]

Thanks for our answer!

I understand your point regarding the GPU usable.

Note: for the X11, it seems to work, but some distro don't distribute the SDL2 libs with xshape support (archlinux if my tests are corrects, based on the 'trollface' test), which makes the test looks like not shaped.

[Kontrabant]

Note that, while it is supported under real X11, trying testshape.c under XWayland just prints:

X Error of failed request:  BadMatch (invalid parameter attributes)
  Major opcode of failed request:  146 ()
  Minor opcode of failed request:  1
  Serial number of failed request:  367
  Current serial number in output stream:  496

and exits, so shape support seems to be absent from XWayland.

[serpilliere]

Hi @Kontrabant Here, on ubuntu and using Xwayland, it's fine.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

[Kontrabant]

Hmm, it seems that opcode 146 on my system corresponds to the "Present" extension, and the minor opcode corresponds to the PresentPixmap request, which SDL doesn't call directly. No clue why the shape test errors under XWayland when every other test works just fine, and it works with real X11. Might be an Nvidia driver bug?

[serpilliere]

Hi @icculus

It does seem like platforms are moving towards having surfaces with alpha channels that will blend with the desktop, because they generally have these, and a compositor, all cooking on the GPU, whether you use them directly or not. But I'm happy to be wrong.

What is the API that uses surfaces with sdl2 in order to shape window now that the shape API is removed?

[slouken]

Take a look at testshape.c, it’s more complex than you need, but will show you how to use transparent windows for a similar effect.

[serpilliere]

Oh! Thank you @slouken !

[slouken]

You’re welcome!

[serpilliere]

Hi @slouken I have tested the example of the main branch, and it seems there is something weird: On the SDL2 branch, the testshape.c is ok using the trollface_32alpha.bmp (given in the SDL2 branch)

But with the main branch, it seems the code 'freezes' in the SDL_RenderPresent function. (using the same picture)

I tested this on Ubuntu 20.04 LTS Note that others examples seems to be ok, even the OpenGL examples and draw examples. (on x11 server)

Am I missing something?

[slouken]

Can you open a new issue for this, for tracking?

Thanks!