MouseEvent positions are incorrectly detected when layers have their own magnification

[davesmith00000]

I've started work on this issue already but I wanted to do a small write up of the problem since it is more complicated than it appears.

The Problem

1. Set your game to magnification 1.
2. Set your layer magnification to 2.
3. Place something clickable like an InputField on the layer.

Observe that mouse detection is done assuming magnification 1, not 2.

Indigo was supposed to handle magnification changes for you so that as a game dev, you just have to assume 1 to 1 and forget that magnification is a thing. That logic was written before layers could be independently magnified, and doesn't work in the context of layers with their own idea of magnification.

Converting events to work with layer magnification and fallback to master magnification is easy enough since they are passed through layer by layer. The snag is InputState, which can also be used to ask if a mouse event has happened and where, but it cannot know what the magnification level is since it lives outside of the notion of layers.

This causes problems for things like ui components that currently use InputState rather than MouseEvents. We could convert the ui components, but that doesn't remove the problem in general.

Some questions:

1. Should InputState account for magnification at all since it will always be wrong unless the whole game is on the same magnification level?
2. Should InputState handle mouse events if they're always going to be wrong (see point 1)
3. Should the bounds method require/optionally allow you to supply a magnification? (Lots of scope for human error here)

[davesmith00000]

After some experimentation and a fair bit of thinking, I've decided this probably isn't as bad as I thought.

Following the principle of least surprise: If someone new to Indigo sets the magnification to 2, they will expect the pixels to be doubled, and for mouse clicks to be accurate to the scale they're looking at, i.e. clicking on a button, clicks the button, at every point on the button.

Therefore, the current behaviour is basically correct.

Using the mouse with layers at a different scale to the global will require more work by the developer.

1. We may be able to provide some assistance to make the working with abnormally scaled layers easier.
2. We should document the behaviour as expected and explain it a bit
3. We should recommend that best practice is probably to set the global magnification to be the magnification you want to use your UI at, since this problem only affects mouse interactions at the time of writing.

[aumann]

I've just hit this issue, too. Maybe it would be useful to have some "scaling" method on Mouse that maps all internal positions to another scaling.

That would, for example, allow adding an extension method to Button (in user code), like this:

extension (btn: Button) def updateScaled(mouse: input.Mouse)(using cfg: MyViewConfig): Outcome[Button] =
    val scaledMouse: input.Mouse = mouse.scaledBy(cfg.uiScaleFactor)
    btn.update(scaledMouse)

[davesmith00000]

Thank you for the thought, I agree. There are basically three touch points I think:

1. Events (e.g. mouse click)
2. Clickable's - Related to events, processing screen elements with event handlers
3. InputState which is generated from events but acts like a look up

In terms of easiness to solve for each of the above:

1. Events - which layer was the thing that processed the event on? No idea, was it even a thing or just some abstract model code - WHAT DOES IT EVEN MEAN?!? 😄
2. Clickable things - This one is easy, we actually do know what layer the clickable thing was one - phew! One down.
3. InputState - "Hey, did anyone click on this box I'm interested in?" "Depends, which layer is it on?" "How should I know?" "..." "..."

Maybe point 2 should just be done regardless actually, easy enough and again is probably the least surprising thing, i.e. I gave a thing a view based event handler and it can handle the events in the view even though I changed the magnification... 🤔

Otherwise, guessing what to do seems hard, and something like the solution you are suggesting where responsibility is given to the programmer but we provide some scaling methods feels more plausible.

I'm going to get the current work out the way and come back to this - if only to give my brain time to chew over the problem in the background.

Unrelated, Scala 3 eh? Nice. :-)

It's interesting that you've laid out your example code like that. So far I've resisted doing any "fancy" API's for lots of reasons. Scala 3 does seem to invite them more though, but that's probably a discussion for another thread.

Thanks again, always helps to bounce ideas around.

[aumann]

Scala 3 does seem to invite them more though, but that's probably a discussion for another thread.

Playing around with Scala 3, I find it incredible just how much more natural/easy it feels to use (using X) over the old (implicit x: X) even though it's just another word in this case. But you're right, that's another topic.