Add blurred Element support to `OutputDamageTracker`

[maxheuer]

The problem

I'm working on getting blurred backgrounds for windows to work with smithay on my toy compositor, but have had some trouble when trying to integrate with the damage tracking in smithay. From my POV adding this feature would be pretty important, but I could also understand if it's outside the scope of the module or too complicated to implement. It seems niri is looking to implement blur though: yalter/niri#54 and this is clearly a roadblock to that.

Rambling about a potential solution

My Implementation

I'll start by explaining how I've approached the blur problem: I have a custom WaylandSurfaceRenderElement that modifies the code to render with rounded borders as well as gaussian blur. To blur I first blit the current render target into a texture and then blur the parts I care about.

All of this worked fine w/o damage tracking but with damage tracking only the pixels directly underneath the damaged elements are redrawn, so when the blur window is moved the edges will try to access parts of the texture that have not been re-rendered into the current target, so edges appear darker after being damaged.

Further, eventually this will clearly run into the problem that when there is damage beneath a blurred window its contents won't correctly update. For example: if a square beneath a blurred window changes from red to green, only the pixels directly above that square will re-render, so if the blur is in a 9px square, parts of the blurred window in the 9px border around the damaged square will be incorrect.

Ideas

• First, there will have to be a way to tell blurred elements from non-blurred elements. The way I've implemented this is to just add a diffuse function to the Element trait (diffuse meaning the size of the blur). I haven't thought about this part too much because I've been working in my local fork, but it is clearly important.
• Second: as the current damage algorithm keeps track of opaque areas, it will now have to keep track of these blurred areas and damage/redraw elements accordingly

Observations

The current damage tracking algorithm has a Vec<Rectangle> to keep track of damage (ignoring at what height the damage is). This is fine because it doesn't need to know the z-index of the damage: if there is damage under a semi-transparent element, it will redraw in that exact same damage area. (with blur this is more complicated)

It does, however need to know the z-index of any opaque regions in order to not cut out any elements on top of opaque regions.

Another oddity: my first idea was to just force each element underneath a blur element to redraw in a 17px by 17px (for 8px blur) square around the damage, but when the buffer is submitted that is actually drawn to the screen even though these redrawn areas aren't part of the damage. Why is that? How does the GbmBofferedSurface know which parts of the buffer to draw to the screen? Am I thinking about this wrong?

Any thoughts are appreciated!

[midischwarz12]

Any further progress on this?

[codecnotsupported]

Bump, did you make any progress?

[bndlfm]

Is it possible to offer a bounty for this? Blurred window support is a must for me.

[midischwarz12]

@maxheuer Are you still alive?

[maxheuer]

It seems there's more interest in this issue now! (cool to see that Niri has users!)

My initial motivation here was for my own little compositor, however I had decided blur as a blocker to getting anything working was misguided, and dropped the work. Looking at Niri, however, it seems it is a functional compositor that knows what it's doing, hence the renewed interest here. Honestly, I haven't worked on my compositor in ~1 year. However, I recently started using KDE Plasma and it reminded me why I liked more minimalist compositor projects. That being said, I might look into picking this up again, depending on how my motivation is (no promises!).

The way I see it, the problem looks like this:

• Niri, sensibly in my opinion, relies on DrmCompositor
• DrmCompositor does damage-tracking automatically with OutputDamageTracker
• Smithay uses RenderElements everywhere that mostly limit what information can be passed around (there is no blur field in a RenderElement, so OutputDamageTracker doesn't have that information to work with)

Basically, DrmCompositor was written with the goal of creating a simple, traditional stacked desktop experience. To implement blur would mean creating an alternate BlurCompositor that does similar things but takes blur into account in its damage-tracking and rendering. It would also necessitate that the BlurCompositor is aware of what elements blur or not, which might be as simple as taking two separate arrays of elements to render: one blurred, one not. And lastly, I imagine niri has already implemented this, but the convenient WaylandSurfaceRenderElement needs to be extended for blur and rounded corners (at least, as I remember, this was the solution I had implemented).

Of course this BlurCompositor would bring up issues of who maintains what code and where... but hacking together a simple demo might be relatively simple (knock on wood), first without damage tracking, and then incrementally adding simple support for damage tracking.

Would love to hear thoughts on this! (and maybe try out Niri for myself...)

[YaLTeR]

Hm, is it really necessary for a whole separate compositor? Maybe just some function on the render element "btw please expand the redrawing rectangle below by X pixels"?

(haven't looked deep into how damage calculation works)

[nferhat]

Not a perfect solution by any means, but there's something going on in

https://github.com/nferhat/fht-compositor

See src/renderer/blur, you add a BlurElement below a element you want the background of blurred.

It does some dirty hacks to bind back buffers (using glGetIntegerV), optimally needing a function in smithay to bind back, etc.

Also after I forgot which PR merged I need to add proper damage tracking rectangles for the blur regions. For now it damages everything in the BlurElement physical region (expanded based on blur radius/passes/whatev)

Hope this is a good path towards a better implementation

[psi4j]

Not a perfect solution by any means, but there's something going on in

https://github.com/nferhat/fht-compositor

See src/renderer/blur, you add a BlurElement below a element you want the background of blurred.

It does some dirty hacks to bind back buffers (using glGetIntegerV), optimally needing a function in smithay to bind back, etc.

Also after I forgot which PR merged I need to add proper damage tracking rectangles for the blur regions. For now it damages everything in the BlurElement physical region (expanded based on blur radius/passes/whatev)

Hope this is a good path towards a better implementation

How has this implementation been working for you? The blur looks smooth in your video, but how does it work day to day?

[nferhat]

Not a perfect solution by any means, but there's something going on in https://github.com/nferhat/fht-compositor See src/renderer/blur, you add a BlurElement below a element you want the background of blurred. It does some dirty hacks to bind back buffers (using glGetIntegerV), optimally needing a function in smithay to bind back, etc. Also after I forgot which PR merged I need to add proper damage tracking rectangles for the blur regions. For now it damages everything in the BlurElement physical region (expanded based on blur radius/passes/whatev) Hope this is a good path towards a better implementation

How has this implementation been working for you? The blur looks smooth in your video, but how does it work day to day?

It works fine I guess, since I mostly do editor work and browsing the web, and blur is disabled on the workspace I play games in, so overall good

It was able to run totally fine with with a integrated Radeon graphics device on a 2k monitor at 120fps, thats not really scientific and I'd rather profile everything, but idk how to-do GPU profiling (and smithay lacks support as of right now) so until then I can't comment much about numbers

Right now trying to improve it with proper damage tracking

[psi4j]

I appreciate you sharing your results with us. Hopefully we can get an update from Smithay sometime in the near future. In the meantime, I've been playing around with attempting to port your blur implementation to niri just to see how that might work. There's also this:

https://gitlab.freedesktop.org/wayland/wayland-protocols/-/merge_requests/272

[YaLTeR]

In the meantime, I've been playing around with attempting to port your blur implementation to niri

Someone already did it fyi: https://github.com/YaLTeR/niri/pull/1634

[psi4j]

In the meantime, I've been playing around with attempting to port your blur implementation to niri

Someone already did it fyi: YaLTeR/niri#1634

Awesome!