Some keyboard commands fail silently and often unpredictably on macOS

[agentofuser]

Hi, I'm starting to play with autopilot, trying to issue some common keyboard commands like switching from terminal to browser with Command+Tab, opening a new browser tab with Command+t, etc., and I'm having difficulty understanding what is happening.

I'm very new to Rust too, so I apologize if I'm doing something silly. These are some of the results I've had so far running a sample binary with cargo run from the command line:

use autopilot::key::*;

fn sleep(n: u64) {
    std::thread::sleep(std::time::Duration::from_millis(n * 1000));
}

fn main() {
    // this works
    type_string("hello world", &[], 0.0, 0.0);

    // expected: HELLO WORLD
    // actual: nothing happens
    sleep(4);
    type_string("hello world", &[Flag::Shift], 0.0, 0.0);

    // expected: _
    // actual: nothing happens
    sleep(4);
    tap(&Character('-'), &[Flag::Shift], 0, 0);

    // this works (Command+Tab to browser)
    tap(&Code(KeyCode::Tab), &[Flag::Meta], 0, 0);
    tap(&Code(KeyCode::Return), &[], 0, 0);

    // expected: open new tab (Command+t)
    // actual: nothing happens
    sleep(4);
    tap(&Character('t'), &[Flag::Meta], 0, 0);

    // this works (Command+Tab back to terminal)
    sleep(4);
    tap(&Code(KeyCode::Tab), &[Flag::Meta], 0, 0);
    tap(&Code(KeyCode::Return), &[], 0, 0);

    // this works
    sleep(4);
    tap(&Code(KeyCode::Backspace), &[], 0, 0);
    tap(&Character('D'), &[], 0, 0);

    // this doesn't work (trying to leave fullscreen mode)
    sleep(4);
    tap(&Character('f'), &[Flag::Meta, Flag::Control], 0, 0);
}

Some calls fail sometimes, but work depending on context. The Command+t (new tab) call, for instance, works occasionally, depending on whether there is some type_string() call before it or some other combination which I haven't been able to reproduce reliably.

When a call to tap() or type_string() fails, there's no error message produced. Is there another way I could look into what's happening?

Thank you for publishing this crate!

[Joni-Aaltonen]

I'm seeing this as well with the modifier flags. Have tried multiple different delay combinations.

Edit: Trying to achieve paste on a mac

autopilot::key::tap(&Character('v'), &[autopilot::key::Flag::Meta], 0, 0);

[ThetaSinner]

Hi @agentofuser and @Joni-Aaltonen, I have an explanation for you as to why the operations you're attempting fail.

The char_to_key_code function https://github.com/autopilot-rs/autopilot-rs/blob/d659a7429ef4f6d12b88ab55c0f47e622c3ae5cf/src/key.rs#L189 returns 0 for many inputs. e.g. the c and v that you would be relying on for copy and paste.

The system_toggle function for macos then skips doing anything https://github.com/autopilot-rs/autopilot-rs/blob/d659a7429ef4f6d12b88ab55c0f47e622c3ae5cf/src/key.rs#L420 The silent failure referenced by this issue!

I have a solution that works for me on my fork. For copy and paste like @Joni-Aaltonen was trying to achieve.

@msanders I'd be happy to work on contributing this fork but I'm pretty new to macos development so I might need a few pointers.

Anyway, I hope this is useful to somebody - I know it's an old issue but you never know :)