Would there be a way to cancel the attempt to 'repopulate the movement array'?

[FindZach]

For my situation it just ends up spamming

'WARN c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel.'

Isn't there a way we could just reset the mouse location and continue on? or just continue on with the problems at the users risk; it is a warning notification after all.

If I restart the script it still uses the last mouse coordinates and it works again just fine; it just gets bottlenecked by attempting to fix this problem; when in reality I will direct the movement to fix itself.

[JoonasVali]

I don't quite understand the use case, as the mouse really should end up on the target pixel and not get lost in the way there. If this is related to the other issue, maybe multiple mousemotionfactories will solve the problem, where you use the one for real screen to 'step out' of the virtual screen.

To answer the question, I don't think there is an api to stop the behavior, you can attempt to use reflection api to change the behavior.

[FindZach]

If it goes off the virtual screen it gives this problem. I don't want the mouse to have any control of my main mouse; it doing its own thing inside of my program.

[JoonasVali]

If it goes off the virtual screen it gives this problem

Why does it attempt to go off the virtual screen?

[FindZach]

If it goes off the virtual screen it gives this problem

Why does it attempt to go off the virtual screen?

because I want it to act as if the mouse is leaving the screen for the intention of making the mouse movements look human.

I have several instances of this program running so it wouldn't make sense for me to have several mouses per instance. Everything would work 100% fine if I dont get stuck in that while loop

[JoonasVali]

Hi, I figured I'll try to understand if this is still an issue.

Could you clarify again what is the problem.

What I understood:

You are running multiple instances of the program (Each one having their own "mouse" inside their screens, independent from your real OS mouse.)

What confuses me:

How can a virtual mouse leave their 'screen' or what's exactly expected to happen in that case? (I understood you don't want to involve your real mouse, so I don't really get what's going on.) You may have a very valid use case, I just don't understand it yet. I don't want to make changes to the library before I fully understand the issue.

[dginovker]

Hey @JoonasVali - We're having the same issue at our project. If you want to reopen the issue, I think more people might be able to provide more insight.

Basically, we put a game inside a window and use the NaturalMouseMotion API to click things. Sometimes our internal API tells the NaturalMouseMotion to go to position (i.e. 0, 44), which is outside the bounds of our window. This gets the API stuck in a perpetual loop of

2022-02-14 04:52:35 [Script] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (0 -> 0) y: (52 -> 44) 
# More lines like above, slightly varying y values..
2022-02-14 04:52:35 [Script] WARN  c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel.

While we can attempt to solve it externally, it's an infinite loop that bounces between the following stack traces internally:

at com.github.joonasvali.naturalmouse.api.MouseMotion.move(MouseMotion.java:198)
at com.github.joonasvali.naturalmouse.api.MouseMotion.move(MouseMotion.java:71)
at com.github.joonasvali.naturalmouse.api.MouseMotionFactory.move(MouseMotionFactory.java:45)

at com.github.joonasvali.naturalmouse.api.MouseMotion.move(MouseMotion.java:177)
at com.github.joonasvali.naturalmouse.api.MouseMotion.move(MouseMotion.java:71)
at com.github.joonasvali.naturalmouse.api.MouseMotionFactory.move(MouseMotionFactory.java:45)

(Bottom one repeats 8 times, then goes back to the top, and the cycle continues..)

It's 5am for me right now so I don't have a test-case or anything but hopefully this makes sense. Alternatively if you re-open the issue another user might notice it again

[dginovker]

One solution to this might be to throw an exception if it detects being stuck in a loop trying to move somewhere impossible

[JoonasVali]

Thanks, I reopened the issue and will try to find time to look into it soon. If you have a reproducing program, then that would be appreciated.

[dginovker]

Interestingly the following code didn't trigger it:

import com.github.joonasvali.naturalmouse.api.MouseMotion;
import com.github.joonasvali.naturalmouse.api.MouseMotionFactory;

public class Demo {
    public static void main(String[] args) throws InterruptedException {
        MouseMotionFactory mouseMotionFactory = MouseMotionFactory.getDefault();
        int yDest = 0;
        int xDest = 55555;
        MouseMotion motion = mouseMotionFactory.build(xDest, yDest);
        motion.move();
        System.out.println("Complete");
    }
}

So maybe it's not quite something as simple as moving it out of bounds? I'm using @gigiaj's https://github.com/OSRSB/OsrsBot, I've pinged him maybe he can chime in on what his wrapper could be doing here

[dginovker]

I'm able to reproduce it in GigiaJ's wrapper with just

mouse.move(259, 0);
System.out.println("Moved to 259, 0");  // Never gets called, is stuck on line above

So I think it has to do with the setup process in some way

Edit: It only happens sometimes though.. Could it have to do with the way the mouse moves to the location..?

---

Edit: Working logs -

Starting move
Inside Mouse.java move with parameters 10 259 0 0 0 0
In mouseMove with parameters 259 0 0 0
Wind mouse to 259,0
MotionFactory to 259,0
2022-02-14 18:59:29 [Script-Goblin Killer] INFO  c.g.j.naturalmouse.api.MouseMotion - Starting to move mouse to (259, 0), current position: (0, 0)
2022-02-14 18:59:30 [Script-Goblin Killer] INFO  c.g.j.naturalmouse.api.MouseMotion - Mouse movement to (259, 0) completed
Moved mouse to 259, 0

Starting move
Inside Mouse.java move with parameters 10 259 0 0 0 0
In mouseMove with parameters 259 0 0 0
Wind mouse to 259,0
MotionFactory to 259,0
2022-02-14 19:00:13 [Script-Goblin Killer] INFO  c.g.j.naturalmouse.api.MouseMotion - Starting to move mouse to (259, 0), current position: (0, 0)
2022-02-14 19:00:13 [Script-Goblin Killer] INFO  c.g.j.naturalmouse.api.MouseMotion - Mouse movement to (259, 0) completed
Moved mouse to 259, 0
Moved to 259, 0
2022-02-14 19:00:13 [Script-Goblin Killer] INFO  rsb.script.Script - Script stopped.

Non-working logs -

Starting move
Inside Mouse.java move with parameters 10 259 0 0 0 0
In mouseMove with parameters 259 0 0 0
Wind mouse to 259,0
MotionFactory to 259,0
2022-02-14 19:01:08 [Script-Goblin Killer] INFO  c.g.j.naturalmouse.api.MouseMotion - Starting to move mouse to (259, 0), current position: (0, 0)
2022-02-14 19:01:09 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (253 -> 275) y: (0 -> 0) 
2022-02-14 19:01:10 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (253 -> 259) y: (0 -> 0) 
2022-02-14 19:01:10 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel.
2022-02-14 19:01:10 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (253 -> 259) y: (0 -> 0) 
2022-02-14 19:01:10 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel.
2022-02-14 19:01:11 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (253 -> 259) y: (0 -> 0) 

Starting move
Inside Mouse.java move with parameters 10 259 0 0 0 0
In mouseMove with parameters 259 0 0 0
Wind mouse to 259,0
MotionFactory to 259,0
2022-02-14 19:04:17 [Script-Goblin Killer] INFO  c.g.j.naturalmouse.api.MouseMotion - Starting to move mouse to (259, 0), current position: (0, 0)
2022-02-14 19:04:17 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (0 -> 259) y: (0 -> 0) 
2022-02-14 19:04:17 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel.
2022-02-14 19:04:18 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (270 -> 279) y: (0 -> 0) 
2022-02-14 19:04:18 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (270 -> 259) y: (0 -> 0) 
2022-02-14 19:04:18 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel.
2022-02-14 19:04:18 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (270 -> 259) y: (0 -> 0) 
2022-02-14 19:04:18 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel.
2022-02-14 19:04:19 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (270 -> 259) y: (0 -> 0) 
2022-02-14 19:04:19 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel.
2022-02-14 19:04:19 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (270 -> 259) y: (0 -> 0) 
2022-02-14 19:04:19 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel.
2022-02-14 19:04:19 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (270 -> 259) y: (0 -> 0) 
2022-02-14 19:04:20 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel.
2022-02-14 19:04:20 [Script-Goblin Killer] WARN  c.g.j.naturalmouse.api.MouseMotion - Mouse off from step endpoint (adjustment was done) x: (270 -> 259) y: (0 -> 0) 

Can't figure out what's happening differently between these runs.. It's not based on starting mouse coords either

[dginovker]

Here's code that copies OSRSBot - Based on its implementation of "hopMouse" (not shown directly since it references another class that's too big to paste), I think it very could well be the hopMouse override is malfunctioning:

import com.github.joonasvali.naturalmouse.api.MouseMotionFactory;
import com.github.joonasvali.naturalmouse.api.SystemCalls;
import com.github.joonasvali.naturalmouse.support.*;
import com.github.joonasvali.naturalmouse.util.FactoryTemplates;

import java.awt.*;
import java.util.Random;

public class Demo {
    public static void main(String[] args) throws InterruptedException {
        MouseMotionNature nature = new MouseMotionNature();
        
        nature.setSystemCalls(new RSBSystemCalls(inputManager));
        
        nature.setDeviationProvider(new SinusoidalDeviationProvider(10.0D));
        nature.setNoiseProvider(new DefaultNoiseProvider(2.0D));
        nature.setSpeedManager(new DefaultSpeedManager());
        nature.setOvershootManager(new DefaultOvershootManager(new Random()));
        nature.setEffectFadeSteps(15);
        nature.setMinSteps(10);
        nature.setMouseInfo(new DefaultMouseInfoAccessor());
        nature.setReactionTimeBaseMs(20);
        nature.setReactionTimeVariationMs(120);
        nature.setTimeToStepsDivider(8.0D);

        MouseMotionFactory motionFactory = FactoryTemplates.createAverageComputerUserMotionFactory(nature);
        motionFactory.setMouseInfo(() -> new Point(0, 0));

        int yDest = 0;
        int xDest = 500;
        for (int i = 0; i < 1000; i++) {
            motionFactory.move(xDest + i * 3, yDest);
            System.out.println("Complete");
        }
    }

    public class RSBSystemCalls implements SystemCalls {

        InputManager inputManager;
        public RSBSystemCalls(InputManager inputManager) {
            this.inputManager = inputManager;
        }

        @Override
        public long currentTimeMillis() {
            return System.currentTimeMillis();
        }

        @Override
        public void sleep(long time) throws InterruptedException {
            Thread.sleep(time);
        }

        @Override
        public Dimension getScreenSize() {
            return Toolkit.getDefaultToolkit().getScreenSize();
        }

        /**
         * <p>Moves the mouse to specified pixel using the provided Robot.</p>
         *
         * <p>It seems there is a certain delay, measurable in less than milliseconds,
         * before the mouse actually ends up on the requested pixel when using a Robot class.
         * this usually isn't a problem, but when we ask the mouse position right after this call,
         * there's extremely low but real chance we get wrong information back. I didn't add sleep
         * here as it would cause overhead to sleep always, even when we don't instantly use
         * the mouse position, but just acknowledged the issue with this warning.
         * (Use fast unrestricted loop of Robot movement and checking the position after every move to invoke the issue.)</p>
         *
         * @param x the x-coordinate
         * @param y the y-coordinate
         */
        @Override
        public void setMousePosition(int x, int y) {
            inputManager.hopMouse(x, y);
        }
    }
}

[dginovker]

Yep - Can confirm in my case it was the setSystemCalls that was causing the issue, and the implementation (provided by) me was malfunctioning.

I think a few things could help if you want to improve this for users -

• WARN c.g.j.naturalmouse.api.MouseMotion - Re-populating movement array. Did not end up on target pixel. should mention it's target pixel and where it is currently. The other log line says it's going to move the mouse, but the mouse movement was the part that was failing, so it can't be taken at face value
• A loop detection and throwing exception could still be helpful - i.e. if it doesn't end up at the target pixel 5x in a row, coming from the same starting location, throw an exception to return to the developer. Add a message like "Mouse failed to go from SOURCE to DEST 5 times - If you overwrote the system calls, check to make sure they're implemented correctly"

Anyways, thanks for the library!

[JoonasVali]

Good suggestions, I'll analyze it and improve accordingly. Thanks!

[dginovker]

@JoonasVali I'm actually getting this issue in a separate situation, not 100% sure of the cause. A loop detection would definitely be nice as a catch-all solution against scripts getting stuck and never being called back from the API.

Could be implemented with something as simple as checking the stack to see if the last 20 methods called are all from within NaturalMouseMotion API and are repeating

[JoonasVali]

@dginovker Thanks, and sorry for not answering earlier. The war in Europe keeps my mind busy in other ways. If you can find a time to create a small example snippet would be great, but otherwise I'll try to find time to look into it anyway at some point.