Mode combination

[nnra6864]

No clue how difficult this would be to implement, but I think it'd be real nice if you could combine say rotate with stretch. Good luck if you decide to code this!

[VirtCode]

I see, it would be fun to combine stuff with stretch.

But as it currently stands this is not really done that easily with the current plugin architecture. I might implement this when I'm refactoring anyways. Thanks for suggesting this.

[minec-aw]

The readme states that you have done the code refactoring, so are you still thinking of implementing this feature? @VirtCode

[VirtCode]

weeelll, the architecture mentioned here is already the "refactored" one, but I'll see about it

[C0Florent]

Well I'm coming here from #95 since you mentioned this would be good to get done first

I just had a little read through of the code, thankfully it's it quite easy to grasp since the project is quite concise and has such a limited scope; and I think it wouldn't be too hard to implement mode combination.

You already have an interface for modes, that's good, we would just have an array of modes instead of a single one in CDynamicCursors, now the question would be how to combine several SModeResults.

I'm not sure it would work well with the current definition of SModeResult, I feel like combining arbitrary rotations and stretching could go wrong (stretching before/after the rotation gives a different result or other combinations idk). I can't resist thinking about transformation matrices for this. In practice they should work seamlessly, though I'm not convinced they're necessary or better in any way (other than design elegance) than your current method.

So here's that, I think this is workable, this is my current thought process for this, I feel like I could do it if you don't have the time right now. I'd like to know your thoughts on the design to implement this, perhaps if you already had some ideas different to what I explained above.

Cheers

[VirtCode]

This is pretty much what I was thinking. The current SModeResult is just the result of incremental growth and should be a transformation matrix as you pointed out. In the end we'd multiply the matrices of all active modes (ideally respecting the order the user has configured) and render with that transformation.

There are a couple of thoughts of problems though which will arise if we switch to an implementation just using transformation matrices. They are not really deal breakers but just things that have stopped me from implementing it previously:

• We generally want to reduce how often we need to render the cursor as much as possible, especially when using hardware cursors as one advantage is that we only need to render the cursor if the shape changes, so not when it moves. So back in the day I implemented a threshold which the rotation had to change in order for the plugin to redraw the cursor (see here. This would of course no longer be possible with matrices and we will have to reduce this to an equality check. But to be honest, I doubt that the threshold at reasonable values did increase performance in any significant way so we can just nuke that IMO.
• When scaling the cursor beyond 1, the plugin needs to be able to detect that and fall back to software cursors if the user is using hardware cursors, see here. In order to be future proof (ideally we use transformation matrices for everything, like cursor magnification), we'd need to somehow detect if the current transformation would scale the shape to be much larger. Maybe we can just transform a couple of points and see whether they fit inside a certain area.
• When rendering as software cursors, we need to come up with a reasonable damage region. Given a scale and rotation this is relatively easy, but we might have to pull a similar trick as above for arbitrary matrices.
• Translation will have to be handled as a separate parameter and not via the transformation matrix. This is because with hardware cursors we cannot translate the shape inside the hardware buffer but will need to move the buffer itself. At the moment, no mode uses translation so this is just something we need to keep in mind.

Given that this will be quite a large refactoring, we might also want to consider the following, which just came to mind:

• We might still want modes to return a struct containing Mat3x3 along other values, mainly opacity as preparation for #95.
• The rendering of this plugin in renderer.cpp is basically a copy of renderTextureInternal in Hyprland which takes a couple more transformation (e.g. stretch vector and angle). Maybe there is a method we could pass our matrix to the original method and could get rid of renderer.cpp alltogether and wouldn't have to use our own renderpass (for sw cursors). Just a thought though, don't know whether that's actually possible without too much hackery.
• It would be clean to also migrate the shake to find to a transformation matrix instead of just a scale I think.

As for the plugin configuration, I think we should make mode accept multiple, space separated, modes. The order should be preserved as e.g. stretch rotate and rotate stretch will produce different results. Also, don't forget to update the shape rule parser so it shaperules support multiple modes.

So yeah that's about my perspective on that. I am all for the changes you proposed, but I just wanted to highlight some things we'll need to consider in order for it to work out in the end. On first glance it sounds really simple and incredibly clean but as you see there are some things to consider which might not be immediately obvious. I hope this huge text wall was not too overwhelming haha.

But if you have the time and motivation feel free to work on this! I'd be more than happy to help if you have any questions, other ideas or issues during the refactoring.

[C0Florent]

Thanks for replying, and don't worry I'm not afraid of text walls, I tend to write my own quite often too.

Good thing we agree on transformation matrices. I don't know if there is already some good implementation we could use or rather implement our own matrix "library". Ideally we would want a good class template for arbitrary numeric types and compile-time size (a template<typename T, size_t width, size_t height>). Custom operators could go a long way to have readable matrix multiplication (with a * that returns a new instance and a *= that assigns to the current instance when the matrix sizes allow it).

I had already noticed there are some checks to avoid re-rendering in some cases, which would probably a bit harder to detect using transformation matrices. I'd be interested to try and implement these checks directly from properties of transformation matrices (if possible without applying the transformation to several points to "see what happens"), good thing I just took a basic linear algebra course. For example I would think (by intuition, I didn't check it) that checking if the matrix scales the cursor to a non-1 factor would involve checking whether its determinant equals 1 (this would combine horizontal and vertical scaling though).

I had also already thought of later bringing opacity as a third dimension to the transformation matrix but this might screw the scale checks I mentioned above, perhaps we'd need to compute the scale checks on a submatrix which only includes screen coordinates then.

Also I might want to learn about the concept of hardware/software cursors, I have no idea what this is about. I'd appreciate if you have some links to useful resources about this.

IIRC there's a trick to include translations in transformation matrices by bringing in an "artificial" additional dimension, which would make them combine with other transformations (rotation then translation differs from the other way around). From what you said about the fact that hardware cursors have some constraints on translation, and I don't know about this, I can't quite be sure.

To compute damage regions I guess the "best" way would be to apply the transformation matrix to the smallest rectangle containing the cursor (transform the four corners of the rectangle) which would give a transformed quadrilateral. I don't know what kind of shapes can be damage regions (whether arbitrary quadrilaterals work), or if we would have to construct some other shape to include this quadrilateral.

As for other things you mentioned (touching renderer.cpp and config options design), I've never had a look at this part of Hyprland, so I don't any meaningful input to provide, I'd have to look into it beforehand.

Short term I'd be more interested to "simply" refactor existing modes to use transformation matrices without implementing mode combination yet (although the API should already make it easy to implement). At least this would leave config options design for later, so it seems like a decent first step.

[VirtCode]

I don't know if there is already some good implementation we could use or rather implement our own matrix "library".

Hyprland already uses its own matrix implementation from hyprutils, but it sadly is nothing to get excited about haha. It only contains 3x3 matrices and does not have operator overloading like you mentioned. But ideally we should still use this as this will play nice with all of hl's functions, albeit at the expense of some readability I guess.

Also I might want to learn about the concept of hardware/software cursors, I have no idea what this is about.

It's actually a quite simple concept (from our perspective, we don't need to know how it is then implemented under the hood). Sadly I don't know of any good resources I could link you to so maybe I should at some point write a blog post myself. But here's a quick rundown which should give you some intuition what hardware cursors are.

The ideal way of doing cursors on modern-ish GPUs is hardware cursors. This means the GPU has a dedicated buffer for the cursor shape that is different from the main framebuffer (where your workspace and windows are drawn to). This dedicated buffer is relatively small and its size depends on the GPU, but is large enough to contain the cursor shape. When outputing frames to the monitor, the GPU will composit the cursor buffer onto the normal framebuffer which is very fast. When the user then moves the cursor the compositor just tells the GPU to move the cursor buffer to a new location, which means we don't have to touch the main framebuffer. We only have to draw into the cursor buffer if the shape changes (or it is transformed by this plugin of course).

Software cursors on the other hand are a more straight-forward and primitive method of rendering the cursor. It just means that we render the cursor shape like any other element (e.g. layer, window, etc.) onto the main framebuffer. When moving the cursor, this is of course not as efficient as just sending a new vec2 to the GPU, as we have to redraw the region of the main framebuffer where the cursor previously was and where it newly is.

Of course there are also a couple of edge cases, mainly Nvidia GPUs not supporting the cursor buffer properly. On Nvidia, we cannot use the GPU (i.e. opengl) to draw into the cursor buffer and hyprland uses cpu rendering to draw into this buffer. Because this is infeasible and inefficient for a plugin which draws into the cursor buffer a ton of times, we just fallback to software cursors there. There are also other scenarios where a compositor should fallback to software cursors, but they don't really matter here.

As for this plugin, we of course have to implement both ways of rendering (see here and here). Because of the physical size constraints of the cursor buffer, we also need to do that size check I mentioned above to resort to software cursors, if our shape would grow larger than the actual cursor buffer.

To get more of a feel for this and if you have a non-nvidia device with this plugin installed and enabled, you can try to enable plugin:dynamic-cursors:hw_debug. This will fill in the entirety of your cursor buffer with a solid color, which changes every time the shape is redrawn. This way you can see for yourself how often it rerenders the shape and how big the buffer is on your specific device:

hyprctl keyword plugin:dynamic-cursors:hw_debug true

IIRC there's a trick to include translations in transformation matrices by bringing in an "artificial" additional dimension

Sounds interesting, would be nice if you can have a look at this. As you probably saw above, the limitation is just that we'd need to be able to get a separate vec2 to translate the cursor buffer. If we just translate during the rendering, we eventually move the shape outside the bounds of the cursor buffer. For software cursors this of course no issue if we compute our damage correctly.

I had also already thought of later bringing opacity as a third dimension to the transformation matrix

To be honest, I don't know whether this is a good idea, as I don't see how it would make things cleaner. Returning an additional scalar for the opacity would be more intuitive and combination is also just multiplication of that scalar. Unless you have some other things in mind which would benefit from it being in the matrix specifically.

that checking if the matrix scales the cursor to a non-1 factor would involve checking whether its determinant equals 1

My intuition of the determinant is a bit limited, but doesn't the determinant tell us more about how space is scaled, and not its bounds? I mean imagine I'd have a transformation which squishes and stretches along the xy diagonal, transforming a unit quad to thing long shape. Wouldn't the determinant then be below 1, even though the top right point would be well beyond (1,1). But honestly if you say this could work out I am all for it, it would be incredibly nice haha.

I don't know what kind of shapes can be damage regions (whether arbitrary quadrilaterals work)

Damage regions have to be axis aligned quads, as hyprland uses OpenGL's scissor test for the damage. But computing this from an arbitrary quadrilateral is quite straight forward.

Short term I'd be more interested to "simply" refactor existing modes to use transformation matrices without implementing mode combination yet

Sounds great!

[C0Florent]

Hyprland already uses its own matrix implementation from hyprutils

Oh ok that's always nice to have. If I end up needing more than this I could contribute over there. Not having operators is not a big deal, I had a look and it has all the functions we should need.

Thanks for the soft-/hardware cursor explanation, it's actually quite simple!

I mean imagine I'd have a transformation which squishes and stretches along the xy diagonal, transforming a unit quad to thing long shape. Wouldn't the determinant then be below 1, even though the top right point would be well beyond (1,1)

Yup, I forgot about this part, you're right. I still think there must be something "smarter" than transforming several points, I'd have to do a tiny bit of research. Our condition would be that the cursor has a scale greater than 1 in any axis-direction, right?

IIRC there's a trick to include translations in transformation matrices by bringing in an "artificial" additional dimension

Sounds interesting, would be nice if you can have a look at this.

Actually I would say the right move for now is to say we'll look into this later because we don't even have a use case for it yet. But if you're curious, that's exactly what the implementation of hyprutils does for translation. Basically, for 2D coords translation, you have to go to 3 by 3 matrices (I guess that's why 3 by 3 is the only implemented size): you multiply by a 3 by 3 matrix similar to an identity matrix, except the translation coordinates are in the third column (in the first and second row).

I had also already thought of later bringing opacity as a third dimension to the transformation matrix

To be honest, I don't know whether this is a good idea, as I don't see how it would make things cleaner. Returning an additional scalar for the opacity would be more intuitive

I mean, yeah that works too of course. We would also be kind of forced to proceed this way if can't choose our matrix size (which will be the case if we use matrices from hyprutils).

Damage regions have to be axis aligned quads

Good to know, should be fine anyway as you said.

I guess I should have a look at the renderer code to know what kind of result I should try to build (basically what the input types are to whatever external functions we call to actually render stuff)

[VirtCode]

Our condition would be that the cursor has a scale greater than 1 in any axis-direction, right?

Weeell that's how it is currently done. Kind-of. As soon as the scale (all axes) is greater than one, we switch to sw cursors. However we don't consider the stretching (from the stretch mode) because there we assume there we don't stretch beyond what the cursor buffer fits, even though that would be beyond 1. Because it would be quite stupid if we'd switch to sw cursors just for a bit of stretching.

I have thought about this for a bit and if we are already digging into this, might I propose the following:

1. Obtain a transformation matrix $A$ from the modes/shake
2. Transform a quad of the size of the cursor shape texture with $A$
3. Check if the transformed quad can fit into the cursor buffer's size, if not fall back to SW
4. Add a translation $\alpha$ to the original matrix $A$ such that all corners of the quad are >= 0
5. Render into the cursor buffer using this matrix $A\alpha$
6. Subtract $\alpha$ from the position where we tell the GPU to place the cursor buffer

This would work with all transformation matrices, but here's an example with a simple rotation:

At the moment, things are a done a bit simpler, as the plugin orignally only rotated the cursor. That mean that it would just translate the shape inside the cursor buffer to a position such that all rotations would fit onto the buffer, so that offset was not different for each frame. But that of course is not entirely correct for the stretch mode but it worked fine enough so I didn't bother. The approach here would be more accurate and would also allow for magnification greater than 1 given they still fit inside the buffer.

But yeah to be fair, this is really a core part of the plugin and not really relevant for #95 so I can also understand if you don't want to dig into this part. You could leave that part as is and botch in a temporary solution to detect whether the shape is magnified, and I'll implement this myself another time if you want.

I guess I should have a look at the renderer code to know what kind of result I should try to build

I think the result for each mode should be a struct containing a Mat3x3 for the transformation and a double for the alpha. Instead of calling projectCursorBox here we'd then just multiply by the matrix received by the mode. The only other thing you'd have to do is handle the cases like sw cursor fallback and damage regions, etc.

The part here "The rendering of this plugin in renderer.cpp is basically a copy..." we can also do another time and I can look into it myself if you are not too familiar with hyprland's rendering. It was more of an anecdote of what I would then do so you don't necessarily have to worry about that.

If you want to you can try some stuff out and draft an MR and I can review, point you to the right things and help out if I find time.

[C0Florent]

I have thought about this for a bit and if we are already digging into this, might I propose the following:

Seems like a good plan for sure!

But yeah to be fair, this is really a core part of the plugin and not really relevant for https://github.com/VirtCode/hypr-dynamic-cursors/issues/95 so I can also understand if you don't want to dig into this part.

I guess only time will tell what I get to do or not. Although my main interest seems like getting #95 done, I don't even have a concrete use case for it yet, I'd have to design/patch a cursor to make it useful. What I kinda mean by that is that I might want to do some work here just because it seems like a fun side project to have, and I've wanted to get into developing Hyprland plugins for a while, knowing I might need to write my own to handle stuff properly for my setup. Also geometry is fun to me, so I'm curious as to how the size checks could be implemented properly.

The part https://github.com/VirtCode/hypr-dynamic-cursors/issues/11#issuecomment-3276472284 "The rendering of this plugin in renderer.cpp is basically a copy..." we can also do another time

Sure thing, I'm all for doing things in separate steps, especially when refactoring such core logic. I'd almost say it's worth having a separate for everything that can be done separately: whenever I feel like being done (or having made significant progress) with one aspect of the refactoring and everything works the same for the end user, that's could be worth creating a PR for.

[VirtCode]

What I kinda mean by that is that I might want to do some work here just because it seems like a fun side project to have, and I've wanted to get into developing Hyprland plugins for a while, knowing I might need to write my own to handle stuff properly for my setup.

Awesome! I also initially started this project because I thought a rotating cursor was funny, and it sounded fun to implement. Don't hesitate to ask if you have any questions or thoughts. You can always open an MR earlier if you encounter any weird issues and would want me to have a look at it. And also, have fun ^^

[enessmr]

I think we could add the , for if more than one mode is selected. Maybe a medium-large-ish refactor but like on implemented on hyprcursor side just implement the , on config c++ side then done i guess 🤔