Add support of keyboard and mouse

[sergystepanov]

WIP This PR adds mouse and keyboard controls for aplicable Libretro cores. As simple as it may sound, the main idea is to capture browser Keyboard and Mouse API events and then feed them in a digestible form to the Libretro frontend. Unfortunately, there are several problems related to the original design choices of cloud-game. Initially, it was assumed that each player would have just one controller device occupying a single controller port in terms of the underlying Libretro API. This way, it would be pretty easy to distinguish controller events of each player, knowing that you can safely forward player control events to their dedicated port without the fear of interfering with other players. However, that was quite a wrong assumption, thinking that one player occupies just one device port. In reality, many cores allow the use of several different controller devices, in theory mapped to an unlimited number of ports (in practice, only up to 9 ports can be used), or even multiplex multiple devices (merge controller events) on a single port, as in the case of the DOSBox Pure core.

Keyboard in Libretro

To use a keyboard with Libretro, two callback functions from the core are required:

1. Keyboard Callback Function: RETRO_ENVIRONMENT_SET_KEYBOARD_CALLBACK

   • Set when the core pushes out the environment variable with the same name.
   • This callback should be called once a user presses or releases a keyboard key.
   • It's important to note that the keyboard controls in Libretro do not have a distinct port argument in the callback function. This limitation makes it impossible to separate player keyboards in cores like DosBox, even though DosBox polls the state of pressed keys for each port separately in the polling callback.
   • The callback function has the following parameters:
     • bool down: Indicates if the key is pressed.
     • unsigned keycode: Internal Libretro representation of the pressed key.
     • uint32_t character: Ignored.
     • uint16_t key_modifiers: Special key modifiers such as shift, alt, ctrl, meta, num/caps/scroll-locks.

2. Device Callback Function: RETRO_DEVICE_KEYBOARD

   • It is polled after each emulation tick in the core_input_state polling function.
   • Each key that was previously pressed by the keyboard callback function must be left in the pressed state or released (by setting the device callback function result to 1 or 0).

Mouse in Libretro

Mouse controls in Libretro are polled similarly to the keyboard, using the core_input_state function with the device name RETRO_DEVICE_MOUSE. The behavior of the callback depends on the id parameter:

• Button States:

  • When the id parameter corresponds to a mouse button, the callback returns the state of that button (pressed or released).
  • In this PR, only the states of the Left, Right, and Middle buttons are implemented.

• Cursor Position:

  • When the id parameter corresponds to the cursor position, the callback returns the position of the mouse cursor.
  • It should be noted that in the case of DOSBox, the X and Y positions of the mouse are not absolute, but rather the relative deltas of X and Y coordinates between two emulation ticks.

Scale cursor speed to native resolution

Since raw pointer movement is not supported in all browsers or platforms, when you stream a display which size is different from a display on which you use mouse, the mouse movement (DPI) should be approximated accordingly. Used algorithm takes coordinates (movement delta) from the virtual display (cursor over the browser video element) and scales them to the default DosBox VGA display size of 640x480. Because the scale ratio value is a floating point number and we can't use subpixel coordinates on the server side, we need to take into account accumulated floating point errors when rounding coordinates to the destination screen (Libretro framebuffer). See an example.

[guozanhua218]

WIP This PR adds mouse and keyboard controls for aplicable Libretro cores. As simple as it may sound, the main idea is to capture browser Keyboard and Mouse API events and then feed them in a digestible form to the Libretro frontend. Unfortunately, there are several problems related to the original design choices of cloud-game. Initially, it was assumed that each player would have just one controller device occupying a single controller port in terms of the underlying Libretro API. This way, it would be pretty easy to distinguish controller events of each player, knowing that you can safely forward player control events to their dedicated port without the fear of interfering with other players. However, that was quite a wrong assumption, thinking that one player occupies just one device port. In reality, many cores allow the use of several different controller devices, in theory mapped to an unlimited number of ports (in practice, only up to 9 ports can be used), or even multiplex multiple devices (merge controller events) on a single port, as in the case of the DOSBox Pure core.

Keyboard in Libretro

To use a keyboard with Libretro, two callback functions from the core are required:

1. Keyboard Callback Function: RETRO_ENVIRONMENT_SET_KEYBOARD_CALLBACK

   • Set when the core pushes out the environment variable with the same name.

   • This callback should be called once a user presses or releases a keyboard key.

   • It's important to note that the keyboard controls in Libretro do not have a distinct port argument in the callback function. This limitation makes it impossible to separate player keyboards in cores like DosBox, even though DosBox polls the state of pressed keys for each port separately in the polling callback.

   • The callback function has the following parameters:

     • bool down: Indicates if the key is pressed.
     • unsigned keycode: Internal Libretro representation of the pressed key.
     • uint32_t character: Ignored.
     • uint16_t key_modifiers: Special key modifiers such as shift, alt, ctrl, meta, num/caps/scroll-locks.

2. Device Callback Function: RETRO_DEVICE_KEYBOARD

   • It is polled after each emulation tick in the core_input_state polling function.
   • Each key that was previously pressed by the keyboard callback function must be left in the pressed state or released (by setting the device callback function result to 1 or 0).

Mouse in Libretro

Mouse controls in Libretro are polled similarly to the keyboard, using the core_input_state function with the device name RETRO_DEVICE_MOUSE. The behavior of the callback depends on the id parameter:

• Button States:

  • When the id parameter corresponds to a mouse button, the callback returns the state of that button (pressed or released).
  • In this PR, only the states of the Left, Right, and Middle buttons are implemented.

• Cursor Position:

  • When the id parameter corresponds to the cursor position, the callback returns the position of the mouse cursor.
  • It should be noted that in the case of DOSBox, the X and Y positions of the mouse are not absolute, but rather the relative deltas of X and Y coordinates between two emulation ticks.

Scale cursor speed to native resolution

Since raw pointer movement is not supported in all browsers or platforms, when you stream a display which size is different from a display on which you use mouse, the mouse movement (DPI) should be approximated accordingly. Used algorithm takes coordinates (movement delta) from the virtual display (cursor over the browser video element) and scales them to the default DosBox VGA display size of 640x480. Because the scale ratio value is a floating point number and we can't use subpixel coordinates on the server side, we need to take into account accumulated floating point errors when rounding coordinates to the destination screen (Libretro framebuffer). See an example.

Could it support virtual keyboard and virtual mouse features which support by retro on mobile? and If run on PC platform, could it supports native mouse and keyboard events?

[guozanhua]

WIP This PR adds mouse and keyboard controls for aplicable Libretro cores. As simple as it may sound, the main idea is to capture browser Keyboard and Mouse API events and then feed them in a digestible form to the Libretro frontend. Unfortunately, there are several problems related to the original design choices of cloud-game. Initially, it was assumed that each player would have just one controller device occupying a single controller port in terms of the underlying Libretro API. This way, it would be pretty easy to distinguish controller events of each player, knowing that you can safely forward player control events to their dedicated port without the fear of interfering with other players. However, that was quite a wrong assumption, thinking that one player occupies just one device port. In reality, many cores allow the use of several different controller devices, in theory mapped to an unlimited number of ports (in practice, only up to 9 ports can be used), or even multiplex multiple devices (merge controller events) on a single port, as in the case of the DOSBox Pure core.

Keyboard in Libretro

To use a keyboard with Libretro, two callback functions from the core are required:

1. Keyboard Callback Function: RETRO_ENVIRONMENT_SET_KEYBOARD_CALLBACK

   • Set when the core pushes out the environment variable with the same name.

   • This callback should be called once a user presses or releases a keyboard key.

   • It's important to note that the keyboard controls in Libretro do not have a distinct port argument in the callback function. This limitation makes it impossible to separate player keyboards in cores like DosBox, even though DosBox polls the state of pressed keys for each port separately in the polling callback.

   • The callback function has the following parameters:

     • bool down: Indicates if the key is pressed.
     • unsigned keycode: Internal Libretro representation of the pressed key.
     • uint32_t character: Ignored.
     • uint16_t key_modifiers: Special key modifiers such as shift, alt, ctrl, meta, num/caps/scroll-locks.

2. Device Callback Function: RETRO_DEVICE_KEYBOARD

   • It is polled after each emulation tick in the core_input_state polling function.
   • Each key that was previously pressed by the keyboard callback function must be left in the pressed state or released (by setting the device callback function result to 1 or 0).

Mouse in Libretro

Mouse controls in Libretro are polled similarly to the keyboard, using the core_input_state function with the device name RETRO_DEVICE_MOUSE. The behavior of the callback depends on the id parameter:

• Button States:

  • When the id parameter corresponds to a mouse button, the callback returns the state of that button (pressed or released).
  • In this PR, only the states of the Left, Right, and Middle buttons are implemented.

• Cursor Position:

  • When the id parameter corresponds to the cursor position, the callback returns the position of the mouse cursor.
  • It should be noted that in the case of DOSBox, the X and Y positions of the mouse are not absolute, but rather the relative deltas of X and Y coordinates between two emulation ticks.

Scale cursor speed to native resolution

Since raw pointer movement is not supported in all browsers or platforms, when you stream a display which size is different from a display on which you use mouse, the mouse movement (DPI) should be approximated accordingly. Used algorithm takes coordinates (movement delta) from the virtual display (cursor over the browser video element) and scales them to the default DosBox VGA display size of 640x480. Because the scale ratio value is a floating point number and we can't use subpixel coordinates on the server side, we need to take into account accumulated floating point errors when rounding coordinates to the destination screen (Libretro framebuffer). See an example.

Could keyboard & mouse version support mouse as touchpad mode ,reference dosbox-pure document

The behavior of a real mouse or touch screen can be controlled by the Input > Mouse Input Mode option.

Virtual mouse (default) (best used when the frontend grabs the mouse input) Direct controlled mouse (not supported by all games) Touchpad mode (drag to move, tap to click, etc., best for touch screens) Off (ignore mouse/touch inputs)

[sergystepanov]

@guozanhua, could you explain how this touchpad mode works and what do you expect from it? Right now, you can press toggle-button under the options not in the fullscreen mode and with that you can touch control mouse movement (virtual cursor) by touching left joystick on the screen.

[guozanhua218]

@guozanhua, could you explain how this touchpad mode works and what do you expect from it? Right now, you can press toggle-button under the options not in the fullscreen mode and with that you can touch control mouse movement (virtual cursor) by touching left joystick on the screen.

I want to cloud-retro work on mobile device or some touch screen device ,can only operator with touch screen like we operate on mobile phone without joystick,The touch screen method is more intuitive, allowing tapping, moving, right clicking, and double clicking at the focal position on the touch screen. The virtual mouse method is a bit cumbersome to operate, but by moving the mouse and then combining it with the joystick to confirm and cancel, it simulates the left and right mouse buttons. The entire operation process has a longer rhythm

[guozanhua218]

@guozanhua, could you explain how this touchpad mode works and what do you expect from it? Right now, you can press toggle-button under the options not in the fullscreen mode and with that you can touch control mouse movement (virtual cursor) by touching left joystick on the screen.

I want to cloud-retro work on mobile device or some touch screen device ,can only operator with touch screen like we operate on mobile phone without joystick,The touch screen method is more intuitive, allowing tapping, moving, right clicking, and double clicking at the focal position on the touch screen. The virtual mouse method is a bit cumbersome to operate, but by moving the mouse and then combining it with the joystick to confirm and cancel, it simulates the left and right mouse buttons. The entire operation process has a longer rhythm

reference this video

and in dosbox_pure_mouse_input setting,there is below information

"dosbox_pure_mouse_input", "Mouse Input Mode", NULL, "You can disable input handling from a mouse or a touchscreen (emulated mouse through joypad will still work)." "\n" "In touchpad mode use drag to move, tap to click, two finger tap to right-click and press-and-hold to drag", NULL, "Input", { { "true", "Virtual mouse (default)" }, { "direct", "Direct controlled mouse (not supported by all games)" }, { "pad", "Touchpad mode (see description, best for touch screens)" }, { "false", "Off (ignore mouse inputs)" }, }, "true"

For your reference

[sergystepanov]

@guozanhua, thanks for the example. I think, on the browser side, touchscreen and mouse should share the same pointer API (which I use for mouse). On the side of nanoarch, it's missing Libretro pointer device API handlers. So, it would need some coding for new pointer controller device in addition to Retropad, keyboard, and mouse. (you can, actually, convert touch events into mouse events on the server for existing API, but that's ugly) OK, maybe I'll add this thing later.

[guozanhua]

@guozanhua, thanks for the example. I think, on the browser side, touchscreen and mouse should share the same pointer API (which I use for mouse). On the side of nanoarch, it's missing Libretro pointer device API handlers. So, it would need some coding for new pointer controller device in addition to Retropad, keyboard, and mouse. (you can, actually, convert touch events into mouse events on the server for existing API, but that's ugly) OK, maybe I'll add this thing later. there is a different between mouse and pointer device, Mouse is for moving position, pointer is for absolute position，reference below code from dosbox_pure_libretro.cpp

[sergystepanov]

@guozanhua, thanks for the example. I think, on the browser side, touchscreen and mouse should share the same pointer API (which I use for mouse). On the side of nanoarch, it's missing Libretro pointer device API handlers. So, it would need some coding for new pointer controller device in addition to Retropad, keyboard, and mouse. (you can, actually, convert touch events into mouse events on the server for existing API, but that's ugly) OK, maybe I'll add this thing later. there is a different between mouse and pointer device, Mouse is for moving position, pointer is for absolute position，reference below code from dosbox_pure_libretro.cpp

Right. I think, it wouldn't matter in case if I'd use normalized coordinates. Initially, I tried to normalize relative coordinates of the mouse, but it didn't work for some reason, and I ditched it because hadn't much time for debugging. But in general, that might work as one API for absolute and relative coordinates. In the Discord channel I wrote:

I think, I'll also switch to normalized coordinates [0; 1] (dx / width, dy / height) for deltas in the server API, this way it will be independent from the screen resolutions and more universal if the code will be merged with morph (libretro uses int16 for x,y positions and the current API as well, but Windows API -- 32bit long, so I'll make them normalized in float32, after that everything may be recalculated on the servers. I haven't tested float precision for int-float-int mapping yet, hopefully it is OK and it should be). (aaaand it didn't work out, lol)