Suggestion: Clipboard

[DrYak]

For sharing clipboard, I've found something that can both be used as a building block for users to make their workaround and which could be used to add clipboard support into lan-mouse:

wl-clipboard

It provides a pair of command that can read from- / write to- the wayland clipboard (and thus be used to store stuff in files, send over pipes or tunnels, etc.)

At the end of the readme, they also mention a rust crate wl-clipboard-rs handling the same which could be used for lan-mouse.

[feschber]

wl-clipboard-rs is definitely a good starting point. It uses the same underlying library as the layer-shell backend in lan-mouse.

There are however some additional problems to solve other than just interacting with the clipboard.

I will just lay out some of my thoughts here for future reference:

1. How to send the clipboard contents to the destination client reliably? (especially larger clipboard contents) (a) Use an additional TCP server socket on each client (b) Chunk the contents into smaller chunks and send them via the same UDP socket.

Option (a) would be a lot easier to implement but I'm definitely leaning towards the second option because it does not require an additional port and protocol. There are however a few issues with it that need solving:

• The clipboard contents must be chunked into a size that fits within UDP packets and later reassembled on the target client.
• Mouse and Keyboard events must not be blocked while clipboard contents are being sent (The mouse should not hang while a 1GiB file is being sent) so this is probably going to require some form of a priority queue (or maybe just a QoS header to let the kernel handle it?).
• Both of the above require changes to the protocol.

2. When to send the clipboard contents? (a) When something is copied on the source client (b) When the cusor enters the destination client (c) When something is pasted on the destination client

I'm also leaning towards the last option here because it would require the least amount of data being sent but I'm also not sure if it is possible to do via standard wayland protocols so I may just resort to option (b) at first.

[DrYak]

Copying the clipboard upon cursor entering client seems indeed the most reasonable.

(And sounds that you almost need to manually implement your own TCP-like layer on top of UDP)

[DrYak]

Hum, maybe #107 could be a temporary solution for the clipboard: just handle it externally of lan-mouse for now (e.g. by using wl-clipboard and the like in external scripts that move the clipboard content around upon moving the pointer to another machine) and so no need to think how to organise transfer of large blocks of data over a piece of software optimised for small and frequent UDP packets.

[feschber]

Hum, maybe #107 could be a temporary solution for the clipboard: just handle it externally of lan-mouse for now (e.g. by using wl-clipboard and the like in external scripts that move the clipboard content around upon moving the pointer to another machine) and so no need to think how to organise transfer of large blocks of data over a piece of software optimised for small and frequent UDP packets.

After giving it some thought this might actually not be the worst idea. Using wl-paste | netcat on device A and netcat | wl-copy on device B would probably work as a hack.

[DrYak]

Using wl-paste | netcat on device A and netcat | wl-copy on device B would probably work as a hack.

Yup! My though also. Specially as wrapping everything in SSH for encryption is also trivial.

wl-paste | ssh env WAYLAND_DISPLAY="wayland-0" ${remote} wl-copy

and attack surface could even be further be reduced by using forced-command and task-specific keys (so for the given ed25519 key used by that script, remote will only ever perform wl-copy and nothing else).

---

edit: environment variable from the actual command

[FireMasterK]

Couldn't the message size limitation be addressed in #104 when implementing DTLS? We wouldn't have any message/packet size issues when using QUIC or DTLS from what I see :thinking:

[feschber]

Couldn't the message size limitation be addressed in #104 when implementing DTLS? We wouldn't have any message/packet size issues when using QUIC or DTLS from what I see 🤔

DTLS is still packet based and those packets are limited in size.

QUIC may be interesting to look into for the future but also does not solve the problem of prioritizing input events over clipboard data.

For now, I want to keep using plain UDP with DTLS.

[feschber]

Using wl-paste | netcat on device A and netcat | wl-copy on device B would probably work as a hack.

Yup! My though also. Specially as wrapping everything in SSH for encryption is also trivial.

wl-paste | ssh env WAYLAND_DISPLAY="wayland-0" ${remote} wl-copy

and attack surface could even be further be reduced by using forced-command and task-specific keys (so for the given ed25519 key used by that script, remote will only ever perform wl-copy and nothing else).

---

edit: environment variable from the actual command

Will check tomorrow if this works

[feschber]

Seems to be working! (order of host and command was swapped)

enter_hook = "wl-paste | ssh thorium env WAYLAND_DISPLAY='wayland-1' wl-copy"

[feschber]

Let me know, if someone can figure out a way of copying files as well ;)

[DrYak]

Tested it with:

enter_hook="wl-paste | ssh argo.fritz.box '-oIdentitiesOnly yes' -i .ssh/id_ed25519_clipboard"

and in the remote's ~/.ssh/authorized_keys:

# forced command
command="/usr/bin/env WAYLAND_DISPLAY='wayland-0' wl-copy",no-agent-forwarding,no-pty,no-X11-forwarding ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDkqOfEBruPi9ceKwIoNS4VqpLX0uEwA0egJPcmMIDMj dryak@saturn

WORKS! YAY!

:confetti_ball: :confetti_ball: :confetti_ball:

When do you plan to tag the release?

[feschber]

The main thing, I wanted to wait for was

• #115, which is now mostly fixed except for #133. (which requires upstream changes in Kwin / xdg-desktop-portal-kde)
• Some bugs I introduced with the internal API change, which is now done
• Input Capture in Windows not working for certain keys.

I think it's probably a good idea to ignore that last part and make a release anyway.

[DrYak]

Addendum:

Use the option wl-paste --no-newline , otherwise wl-paste will automatically add an extra new-line at the end of text type clip board (i.e.: when you copy-paste text as opposed to screen shots), and thus on each screen switch the content of the clip board will get longer and longer as each client adds an extra line when copying this ever growing clipboard.

enter_hook="wl-paste --no-newline | …

[DrYak]

Just a short update: I've updated my OpenSUSE Tumbleweed package to 0.8.0 and are now enjoying the clipboard-over-enter_hook functionnality on several of my machines.

Thanks for the support!

[Kelerion-UK]

This solution still requires external intervention (ie: hooks), right? doesn't work out of the box? is there a timeline for that?

Thanks!

[DrYak]

As feschber has mentionned above, lan-mouse is designed around UDP sending small packets around without guarantee of arrival, which is fine for real-time input sharing (e.g.: 1 packet per mouse event).

Clipboard content can be large and needs to be entirely transmitted, more the realm of TCP.

Direct support would require re-engineering lan-mouse to support an entire separate protocol over TCP just for clipboards or re-implent its own content chunking, resend, etc. (i.e. reinvent a TCP-like on top of the existing UDP).

Separating concerns and having lan-mouse concentrate on input forwarding and let other tech handle clipboard transfers as feschber has currently implemented it, is imho the way to go. What would be still needed is a section in the readme explaining how to get wl-clipboard-over-SSH setup working.

[markmandel]

As a suggestion - you may want to look at game udp protocol libraries for this, as they have solved many of these problems, as they are essentially the same as what you are trying to do.

Some ideas:

• https://crates.io/crates/laminar is probably the most utilised.
• But a KCP implementation might be a fit
  • https://crates.io/crates/kcp
  • https://crates.io/crates/tokio_kcp

[fentas]

Not sure, I Had a look around and found this one https://github.com/songokas/clipboard-sync Worth a look?

[feschber]

Not sure, I Had a look around and found this one https://github.com/songokas/clipboard-sync Worth a look?

Sounds like this could work standalone? I can definitely take some inspiration but will not be working on this until encryption is implemented.