testcafe

testcafe copied to clipboard

What is your Scenario?

I want to run testcafe on a Github Actions macOS 10.15 instance

What is the Current behavior?

Regardless of passing --disable-screenshots and using headless browsers, and not enabling video, testcafe requires the screen recording permission, which there's no clear way to grant from the command line, only the UI

ERROR UnableToAccessScreenRecordingAPIError: The find-window process cannot access the Screen Recording API.
    at runFindWindowBinary (/Users/runner/work/learn-something/learn-something/node_modules/testcafe-browser-tools/src/api/find-window.js:36:19)
    at Object.default_1 (/Users/runner/work/learn-something/learn-something/node_modules/testcafe-browser-tools/src/api/find-window.js:43:15)
    at checkAuthentication (/Users/runner/work/learn-something/learn-something/node_modules/testcafe/src/cli/authentication-helper.ts:28:26)
    at Object.authenticationHelper [as default] (/Users/runner/work/learn-something/learn-something/node_modules/testcafe/src/cli/authentication-helper.ts:43:29)
    at Runner._checkRequiredPermissions (/Users/runner/work/learn-something/learn-something/node_modules/testcafe/src/runner/index.js:530:27)
    at Runner._validateBrowsers (/Users/runner/work/learn-something/learn-something/node_modules/testcafe/src/runner/index.js:304:13)
    at Runner._validateRunOptions (/Users/runner/work/learn-something/learn-something/node_modules/testcafe/src/runner/index.js:461:9)

Type "testcafe -h" for help.

What is the Expected behavior?

When screenshots are disabled and video is not enabled, the screen recording permission should not be required

What is your public website URL? (or attach your complete example)

https://learnsomething.app

What is your TestCafe test code?

n/a, happens with any tests

Your complete configuration file

testcafe --config-file ./testcafe.ci.json safari:headless ./e2e/*.test.ts

testcafe.ci.json:

{
    "disableScreenshots": true
}

Your complete test report

ERROR UnableToAccessScreenRecordingAPIError: The find-window process cannot access the Screen Recording API.
    at runFindWindowBinary (/Users/runner/work/learn-something/learn-something/node_modules/testcafe-browser-tools/src/api/find-window.js:36:19)
    at Object.default_1 (/Users/runner/work/learn-something/learn-something/node_modules/testcafe-browser-tools/src/api/find-window.js:43:15)
    at checkAuthentication (/Users/runner/work/learn-something/learn-something/node_modules/testcafe/src/cli/authentication-helper.ts:28:26)
    at Object.authenticationHelper [as default] (/Users/runner/work/learn-something/learn-something/node_modules/testcafe/src/cli/authentication-helper.ts:43:29)
    at Runner._checkRequiredPermissions (/Users/runner/work/learn-something/learn-something/node_modules/testcafe/src/runner/index.js:530:27)
    at Runner._validateBrowsers (/Users/runner/work/learn-something/learn-something/node_modules/testcafe/src/runner/index.js:304:13)
    at Runner._validateRunOptions (/Users/runner/work/learn-something/learn-something/node_modules/testcafe/src/runner/index.js:461:9)

Type "testcafe -h" for help.
ERROR: "test:e2e:safari:headless" exited with 1.
ERROR: "test:e2e:ci:mac" exited with 1.
Error: Process completed with exit code 1.

Screenshots

No response

Steps to Reproduce

1. Run testcafe on a macOS 10.15 instance in Github Actions

TestCafe version

1.17.1

Node.js version

v15.3.0

Command-line arguments

--config-file ./testcafe.ci.json safari:headless ./e2e/*.test.ts

Browser name(s) and version(s)

Safari, not sure of the version

Platform(s) and version(s)

macOS 10.15 in Github Actions

Other

No response

Hello,

Please update to testcafe v.1.17.2-rc.3 and let us know if the problem has been resolved.

I just tried it with 1.17.2-rc.3 and got the same error as before

Same on 1.18 as well, in case that matters

Hi,

Thank you for your input. We have reproduced the problem. Please stay tuned.

Note for team - reproducible on Intel machine.

I'm still having the same issues on 1.18, already made a ticket a month ago and it's not fixed or answered.

Hi, @joshhansen, this issue occurs because SIP on Github Actions is enabled. Using your own runner with disabled SIP should help.

@roelvandijk88, the problem you mentioned was fixed for local test running in testcafe v1.18.0. Please follow v1.18.0 upgrade instructions. If the issue persists on your local machine after that, please open a new issue with an example project and a video to show how to reproduce the issue. Make sure that the call stack of the problem is also visible.

It's working again with the v1.18.0 upgrade instructions. My studio isn't working but that should be fixed with the release of 1.7.0. Thanks for the help :)

@AlexanderMoiseev Thanks, I understand that SIP is the immediate origin of the error. My point in filing this bug was that, with TestCafe set to neither take screenshots nor record video, screen recording should not be required at all, and SIP should not be triggered to begin with. Since self-hosted runners are costly and the SIP trigger seems needless, it feels like an easy win to get testcafe to provoke the screen record permission only when screen recording is actually needed.

Solved it. it happens when you grant permission to the globally installed testcafe, instead of the one installed in your node_modules. You can grant permissions to the testcafe under: node_modules/testcafe-browser-tools/bin/mac/TestCafe Browser Tools.app it fixed it for me.

I am seeing exactly the same problem in CircleCI with TestCafe 1.18.3... please advise how we can workaround this

For information on how to grant permissions in macOS CircleCI images, please refer to the Setting up a macOS UI Test Project topic. This document contains information about the macOS permission orb that can record Safari automation permissions to the macOS database.

Unfortunately, even if we manage to get through the Screen Capture permission, we still have to deal with the Automation permission. There are more details in a comment here: https://github.com/DevExpress/testcafe/pull/7028#issuecomment-1130559567

I was trying to run TestCafe with Device Farm Mobile browsers https://github.com/DevExpress/testcafe/issues/7026 and noticed that if my testcafe instance is only ever connecting to remote browsers, then it shouldn't need any of the local permissions required by Mac OS. By default, if I tried to spin up testcafe on DeviceFarm it would fail with an error saying I don't have permissions to start the TestCafe Browser Tools app. I ended up just applying the same changes from the pull request in #7028 and that unblocked me.

Would it make sense to have testcafe skip checking permissions if it is being used with the remote flag or a browser-provider that connects to remote devices?

Hi @pterofractal

Would it make sense to have testcafe skip checking permissions if it is being used with the remote flag

TestCafe checks permissions only for the local browsers. So, the permission request dialog doesn't appear when you are using the remote browser. If you see that - it's a bug. In this case, please record a video with the steps required to reproduce the issue.

Fix for CircleCI:

orbs:
  mac-permissions: circleci/[email protected]
commands:
  permit-testcafe-macos:
    description: Allow TestCafe to do whatever it wants
    steps:
        - mac-permissions/add-safari-permissions
        - mac-permissions/add-permission:
            bundle-id: com.devexpress.testcafe-browser-tools
            permission-type: kTCCServiceScreenCapture
        - run:
            name: Allow TestCafe to interact with Safari
            command: |
              safari_events="replace into access (service,client,client_type,auth_value,auth_reason,auth_version,indirect_object_identifier_type,indirect_object_identifier,flags,last_modified) values ('kTCCServiceAppleEvents','com.devexpress.testcafe-browser-tools',0,2,4,1,0,'com.apple.Safari',0,?);"
              sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" "$safari_events"
              sudo sqlite3 "/Users/$USER/Library/Application Support/com.apple.TCC/TCC.db" "$safari_events"
        - mac-permissions/list-permissions

Will almost certainly need CodeSigning changes for newer versions of macOS though (info here)

More details here

Hi @mcmeeking

Thank you for sharing the fix.

Hi,

To workaround this issue on Azure DevOps I'm using this workaround found at Stackoverflow: https://stackoverflow.com/questions/63341991/are-there-any-solutions-to-the-fact-that-testcafe-doesnt-work-on-macos-images-w/63355065#63355065

testcafe remote test.js --hostname ${HOSTNAME} --ports ${PORT1},${PORT2} &
pid=$!
open -a Safari http://${HOSTNAME}:${PORT1}/browser/connect
wait $pid

However when a test fails, it seems that this approach isn't picking up the failure and the job keeps running untill it finally times out after 60 minutes.

Is there a configuration setting or option to disconnect the remote session right away if the test fails?

Hello @thomasvdb By default, Testcafe terminates a session when a test fails. Try running these tests locally. If the issue is reproduced locally, please share an example so that we can reproduce the issue on our side. If the issue can be reproduced only on Azure DevOps, then the issue is related to something else. In this case, we will also need to take a look at an example, as well as the configuration of your AzureDevOps.

I work in a closed secure environment and am not able to grant macOS permission to run screen capture. Any suggestions how to disable screen recording and the os check. My tests used to work on old Mac. I got a new Mac and now I can’t run tests. The Mac is locked and admins are not allowed to modify settings to customize for user.

The feature that prevents test cafe from working when screen recording is not allowed feels wrong. I would benefit from running my tests without screen capture than not being able to run any tests at all.

Hi @javadoug,

Another way exists but it has all limitations of remote tests. Create a remote connection and just run your browser, for example, Safari in this way:

exec(`open -a /Applications/Safari.app ${url}`);

where url - is a value returned by the createBrowserConnection function.

Going to drop this here, would be great if testcafe was able to figure out it's running in github actions and execute this on its own, but hey, there seems to be a way now.

https://github.com/actions/runner-images/issues/7818

Hi @panva,

There are still problems with running tests on the macos-12 GitHub image. The only way now is to run tests on macos-13, but before running them you should execute the following commands:

sudo sqlite3 "$HOME/Library/Application Support/com.apple.TCC/TCC.db" "INSERT OR REPLACE INTO access VALUES('kTCCServiceAppleEvents','com.devexpress.testcafe-browser-tools',0,2,3,1,X'fade0c0000000068000000010000000700000007000000080000001443fa4ca5141baeda21aeca1f50894673b440d4690000000800000014f8afcf6e69791b283e55bd0b03e39e422745770e0000000800000014bf4fc1aed64c871a49fc6bc9dd3878ce5d4d17c6',NULL,0,'com.apple.Safari',X'fade0c000000002c00000001000000060000000200000010636f6d2e6170706c652e53616661726900000003',NULL,1687952810);"
sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" "INSERT OR REPLACE INTO access VALUES('kTCCServiceScreenCapture','com.devexpress.testcafe-browser-tools',0,2,3,1,X'fade0c0000000068000000010000000700000007000000080000001443fa4ca5141baeda21aeca1f50894673b440d4690000000800000014f8afcf6e69791b283e55bd0b03e39e422745770e0000000800000014bf4fc1aed64c871a49fc6bc9dd3878ce5d4d17c6',NULL,0,'UNUSED',NULL,0,1687952810);"

Hi @Aleksey28,

I didn't realize you're behind DevExpress, thank you for sorting this out with the github runners team <3

@Aleksey28 do you have a working TCC.db adjustment for the apple silicon based macos-14 runner?

Hello @panva,

Could you clarify if you tried executing commands from this comment for macOS-14 and share the result you got?

@Bayheck yeah, here you go https://github.com/panva/oauth4webapi/actions/runs/7840958151/job/21396423974#step:6:6

You may test this on your own CI integration by updating https://github.com/DevExpress/testcafe/blob/876db349fe2f588cee70e5eac6562957a29291d8/.github/workflows/test-functional-local-safari.yml#L19 and https://github.com/DevExpress/testcafe/blob/876db349fe2f588cee70e5eac6562957a29291d8/.github/workflows/test-functional.yml#L137-L141

Ah, here's the macos-14 addition

https://github.com/actions/runner-images/blob/e844facbb931ee1a122ef569d76b68edfaf1d8f0/images/macos/scripts/build/configure-tccdb-macos.sh#L35-L38

So for macos-14 it's

sudo sqlite3 "$HOME/Library/Application Support/com.apple.TCC/TCC.db" "INSERT OR REPLACE INTO access VALUES('kTCCServiceAppleEvents','com.devexpress.testcafe-browser-tools',0,2,3,1,X'fade0c0000000068000000010000000700000007000000080000001443fa4ca5141baeda21aeca1f50894673b440d4690000000800000014f8afcf6e69791b283e55bd0b03e39e422745770e0000000800000014bf4fc1aed64c871a49fc6bc9dd3878ce5d4d17c6',NULL,0,'com.apple.Safari',X'fade0c000000002c00000001000000060000000200000010636f6d2e6170706c652e53616661726900000003',NULL,1687952810,NULL,NULL,'UNUSED',1687952810);"
sudo sqlite3 "/Library/Application Support/com.apple.TCC/TCC.db" "INSERT OR REPLACE INTO access VALUES('kTCCServiceScreenCapture','com.devexpress.testcafe-browser-tools',0,2,3,1,X'fade0c0000000068000000010000000700000007000000080000001443fa4ca5141baeda21aeca1f50894673b440d4690000000800000014f8afcf6e69791b283e55bd0b03e39e422745770e0000000800000014bf4fc1aed64c871a49fc6bc9dd3878ce5d4d17c6',NULL,0,'UNUSED',NULL,0,1687952810,NULL,NULL,'UNUSED',1687952810);"