appium-mac2-driver icon indicating copy to clipboard operation
appium-mac2-driver copied to clipboard

Published 20 hours ago • verified publisher icon appium

How to use two modifier keys at once?

Open mythsunwind opened this issue 3 years ago • 6 comments

I try to press Option+Command+f with the custom command extension "macos: keys" on appium@beta and with a custom build java-client (because the 7.4.1 does not contain mac2-driver support):

final int KEY_OPTION = 1 << 3;
final int KEY_COMMAND = 1 << 4;

final Map<String, Object> args = new HashMap<>();
final Map<String, Object> key = new HashMap<>();
key.put("key", "f");
key.put("modifierFlags", KEY_OPTION + KEY_COMMAND);
Object[] keys = {key};
args.put("keys", keys);
getDriver().executeScript("macos: keys", args);

The log shows no error:

2021-03-04 14:13:36:432 [HTTP] {"script":"macos: keys","args":[{"keys":[{"modifierFlags":24,"key":"f"}]}]}
2021-03-04 14:13:36:433 [W3C (976b877a)] Calling AppiumDriver.execute() with args: ["macos: keys",[{"keys":[{"modifierFlags":24,"key":"f"}]}],"976b877a-c45d-4bd8-807d-582447c7fd81"]
2021-03-04 14:13:36:433 [Mac2Driver] Executing extension command 'macos: keys'
2021-03-04 14:13:36:440 [WD Proxy] Proxying [POST /wda/keys] to [POST http://127.0.0.1:10100/session/FE08B980-390F-4D96-BF5D-1FAD9803D8A6/wda/keys] with body: {"keys":[{"modifierFlags":24,"key":"f"}]}
2021-03-04 14:13:36:760 [WD Proxy] Got response with status 200: {"value":null,"sessionId":"FE08B980-390F-4D96-BF5D-1FAD9803D8A6"}
2021-03-04 14:13:36:761 [W3C (976b877a)] Responding to client with driver.execute() result: null

Any idea what is wrong? Do I concatenate the two modifiers correctly (1 << 3 = 8 and 1 << 4 is 16, so i thought 24 might be right)?

mythsunwind avatar Mar 04 '21 14:03 mythsunwind

The W3C spec compatible way of pressing keys through dispatching actions seems to be not supported by the driver:

new Actions(getDriver())
                .keyDown(Keys.ALT)
                .keyDown(Keys.META)
                .sendKeys("f")
                .keyUp(Keys.META)
                .keyUp(Keys.ALT)
                .perform();

results in

2021-03-04 14:25:23:733 [HTTP] --> POST /wd/hub/session/9495a057-8491-457d-82dd-e0964d13c3b2/actions
2021-03-04 14:25:23:733 [HTTP] {"actions":[{"type":"key","actions":[{"type":"keyDown","value":""},{"type":"keyDown","value":""},{"type":"keyDown","value":"f"},{"type":"keyUp","value":"f"},{"type":"keyUp","value":""},{"type":"keyUp","value":""}],"id":"default keyboard"}]}
2021-03-04 14:25:23:734 [W3C (9495a057)] Driver proxy active, passing request on via HTTP proxy
2021-03-04 14:25:23:735 [WD Proxy] Matched '/wd/hub/session/9495a057-8491-457d-82dd-e0964d13c3b2/actions' to command name 'performActions'
2021-03-04 14:25:23:736 [WD Proxy] Proxying [POST /wd/hub/session/9495a057-8491-457d-82dd-e0964d13c3b2/actions] to [POST http://127.0.0.1:10100/session/E3BDAEEA-F0EE-4D85-8597-24752471FE27/actions] with body: {"actions":[{"type":"key","actions":[{"type":"keyDown","value":""},{"type":"keyDown","value":""},{"type":"keyDown","value":"f"},{"type":"keyUp","value":"f"},{"type":"keyUp","value":""},{"type":"keyUp","value":""}],"id":"default keyboard"}]}
2021-03-04 14:25:23:738 [WD Proxy] Got response with status 500: {"value":{"error":"unknown error","message":"Error Domain=io.appium.WebDriverAgentMac Code=1 \"Only actions of '(\n    pointer\n)' types are supported. 'key' is given instead for action with id 'default keyboard'\" UserInfo={NSLocalizedDescription=Only actions of '(\n    pointer\n)' types are supported. 'key' is given instead for action with id 'default keyboard'}","traceback":""},"sessionId":"E3BDAEEA-F0EE-4D85-8597-24752471FE27"}
2021-03-04 14:25:23:775 [W3C] Matched W3C error code 'unknown error' to UnknownError

In short Only actions of 'pointer' types are supported 😢

mythsunwind avatar Mar 04 '21 14:03 mythsunwind

hi Sven,

I've just tried your scenario locally and it worked with e2e tests (e.g. the search dialog has been opened for TextEdit):

  it('should send keys with multiple modifiers into a text view', async function () {
    const el = await driver.elementByClassName('XCUIElementTypeTextView');
    await el.click();
    const modifierFlags = 1 << 3 | 1 << 4;
    await driver.execute('macos: keys', {
      keys: [{
        key: 'f',
        modifierFlags,
      }]
    });
    await B.delay(5000);
  });

logs:

info HTTP 
info HTTP --> POST /wd/hub/session/871ca64d-edc8-43d0-b152-279139ac41ef/execute
info HTTP {"script":"macos: keys","args":{"keys":[{"key":"f","modifierFlags":24}]}}
dbug MJSONWP (871ca64d) Calling Mac2Driver.execute() with args: ["macos: keys",{"keys":[{"key":"f","modifierFlags":24}]},"871ca64d-edc8-43d0-b152-279139ac41ef"]
info Mac2Driver Executing extension command 'macos: keys'
dbug WD Proxy Proxying [POST /wda/keys] to [POST http://127.0.0.1:10100/session/F596F594-1C4A-49DD-995E-4193401FA8F6/wda/keys] with body: {"keys":[{"key":"f","modifierFlags":24}]}
dbug WebDriverAgentMac [xcodebuild] t =     3.54s Type 'f' key with modifiers '⌘⌥' (0x18)
dbug WebDriverAgentMac [xcodebuild] t =     3.54s     Wait for com.apple.TextEdit to idle
dbug WebDriverAgentMac [xcodebuild] t =     3.54s     Find the Application 'com.apple.TextEdit'
dbug WebDriverAgentMac [xcodebuild] t =     3.78s     Synthesize event
dbug WebDriverAgentMac [xcodebuild] t =     4.02s     Wait for com.apple.TextEdit to idle
dbug WD Proxy Got response with status 200: {"value":null,"sessionId":"F596F594-1C4A-49DD-995E-4193401FA8F6"}
dbug MJSONWP (871ca64d) Responding to client with driver.execute() result: null
info HTTP <-- POST /wd/hub/session/871ca64d-edc8-43d0-b152-279139ac41ef/execute 200 493 ms - 76
info HTTP

Unfortunately, there is no easy way to implement actions, because XCTest does not provide interfaces to emulate down/up key states

mykola-mokhnach avatar Mar 04 '21 15:03 mykola-mokhnach

Regarding java client - feel free to join Appium slack, #java_client channel and ask maintainers to release a new version there. Btw the version, that includes a class for Mac2Driver will be published tomorrow.

mykola-mokhnach avatar Mar 04 '21 16:03 mykola-mokhnach

With appium:showServerLogs enabled the first part of my output in the logs looks exactly like yours with Mac2Driver v0.11.0 but the app does not respond and i have some pboard server errors:

2021-03-04 17:49:11:910 [HTTP] {"script":"macos: keys","args":[{"keys":[{"modifierFlags":24,"key":"f"}]}]}
2021-03-04 17:49:11:911 [W3C (003f5e0e)] Calling AppiumDriver.execute() with args: ["macos: keys",[{"keys":[{"modifierFlags":24,"key":"f"}]}],"003f5e0e-fd47-4c1d-bca8-44deb713a711"]
2021-03-04 17:49:11:911 [Mac2Driver] Executing extension command 'macos: keys'
2021-03-04 17:49:11:918 [WD Proxy] Proxying [POST /wda/keys] to [POST http://127.0.0.1:10100/session/E977BA97-F6AC-4F58-9CBE-CAB8D84E4172/wda/keys] with body: {"keys":[{"modifierFlags":24,"key":"f"}]}
2021-03-04 17:49:11:920 [WebDriverAgentMac] [xcodebuild] t =     4.63s Type 'f' key with modifiers '⌘⌥' (0x18)
2021-03-04 17:49:11:921 [WebDriverAgentMac] [xcodebuild] t =     4.63s     Wait for com.wearezeta.zclient.mac to idle
2021-03-04 17:49:11:924 [WebDriverAgentMac] [xcodebuild] t =     4.63s     Find the Application 'com.wearezeta.zclient.mac'
2021-03-04 17:49:11:979 [WebDriverAgentMac] [xcodebuild] t =     4.69s     Synthesize event
2021-03-04 17:49:11:981 [WebDriverAgentMac] [xcodebuild] 2021-03-04 18:49:11.980421+0100 WebDriverAgentRunner-Runner[99687:1727377] [general] Connection to 'pboard' server had an error: <error: 0x7fff9549a9a0> { count = 1, transaction: 0, voucher = 0x0, contents =
2021-03-04 17:49:11:981 [WebDriverAgentMac]     "XPCErrorDescription" => <string: 0x7fff9549ab08> { length = 18, contents = "Connection invalid" }
2021-03-04 17:49:11:981 [WebDriverAgentMac] }
2021-03-04 17:49:11:981 [WebDriverAgentMac] 2021-03-04 18:49:11.980470+0100 WebDriverAgentRunner-Runner[99687:1727114] [general] Failed to set up CFPasteboardRef 'DefaultAsciiKeyboardLayoutPasteboard'. Error: <error: 0x7fff9549a9a0> { count = 1, transaction: 0, voucher = 0x0, contents =
2021-03-04 17:49:11:981 [WebDriverAgentMac]     "XPCErrorDescription" => <string: 0x7fff9549ab08> { length = 18, contents = "Connection invalid" }
2021-03-04 17:49:11:981 [WebDriverAgentMac] }
2021-03-04 17:49:11:982 [WebDriverAgentMac] [xcodebuild] 2021-03-04 18:49:11.981542+0100 WebDriverAgentRunner-Runner[99687:1727114] [general] Failed to set up CFPasteboardRef 'DefaultAsciiKeyboardLayoutPasteboard'. Error: <error: 0x7fff9549a9a0> { count = 1, transaction: 0, voucher = 0x0, contents =
2021-03-04 17:49:11:982 [WebDriverAgentMac]     "XPCErrorDescription" => <string: 0x7fff9549ab08> { length = 18, contents = "Connection invalid" }
2021-03-04 17:49:11:982 [WebDriverAgentMac] }
2021-03-04 17:49:12:226 [WebDriverAgentMac] [xcodebuild] t =     4.93s     Wait for com.wearezeta.zclient.mac to idle
2021-03-04 17:49:12:245 [WD Proxy] Got response with status 200: {"value":null,"sessionId":"E977BA97-F6AC-4F58-9CBE-CAB8D84E4172"}
2021-03-04 17:49:12:245 [W3C (003f5e0e)] Responding to client with driver.execute() result: null

I'm using xcode version 12.3 and appium@beta. The keyboard layout is US.

mythsunwind avatar Mar 04 '21 17:03 mythsunwind

2021-03-04 17:49:11:981 [WebDriverAgentMac] [xcodebuild] 2021-03-04 18:49:11.980421+0100 WebDriverAgentRunner-Runner[99687:1727377] [general] Connection to 'pboard' server had an error: <error: 0x7fff9549a9a0> { count = 1, transaction: 0, voucher = 0x0, contents = 2021-03-04 17:49:11:981 [WebDriverAgentMac] "XPCErrorDescription" => <string: 0x7fff9549ab08> { length = 18, contents = "Connection invalid" } 2021-03-04 17:49:11:981 [WebDriverAgentMac] }

This looks odd. Could be an XCTest bug. You could also try to send keys via AppleScript as a possible workaround

mykola-mokhnach avatar Mar 04 '21 19:03 mykola-mokhnach

Very odd indeed. Actually everything works fine with other key combinations than Option+Command+f

mythsunwind avatar Mar 10 '21 10:03 mythsunwind