Fix the map position jumping if the mouse moves while scrolling

[BudgieInWA]

If the mouse moves at all while scrolling, the map pans with the mouse movement in a very janky way, as shows in this video:

https://user-images.githubusercontent.com/3784591/201303563-7ab133c3-1969-4476-8187-850ce9b49482.mp4

This seems to fix it!