Chromedriver GUI test fails in last macOS arm64 images

[ferpasri]

Description

TESTAR is a testing tool that uses Chromedriver to launch web applications and extract GUI information from the widgets that conform the GUI state. We use GitHub runner images to maintain workflow tests across Windows, Ubuntu, and macOS environments.

Since the release of the macOS-14-arm64 runner image version 20240922.1, the macOS Chromedriver GUI test has started failing. However, this same TESTAR Chromedriver test continues to work on Windows and Ubuntu runner images, and also works correctly on a local macOS laptop with the following specs:

14.6.1
macos 14 that is also arm64
ChromeDriver 130

Given that the Chromedriver test runs successfully on local macOS systems, I would like to know if this failure may be related to recent changes in the macOS-14-arm64 runner images.

Thanks!

Platforms affected

• [ ] Azure DevOps
• [ ] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] Ubuntu 24.04
• [ ] macOS 12
• [ ] macOS 13
• [ ] macOS 13 Arm64
• [ ] macOS 14
• [X] macOS 14 Arm64
• [ ] macOS 15
• [X] macOS 15 Arm64
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

Failing 20240922.1 workflow Failing 20241022.361 workflow

Is it regression?

Working 20240903.5 workflow

Expected behavior

This test should launch chromedriver to open a Chrome GUI instance. Then, the tool should be able to detect GUI widgets and pass the test.

Actual behavior

This test uses chromedriver to open Chrome, but no GUI widgets are detected.

Repro steps

Install Java 11, 17, or 21

1. Clone TESTAR_dev repository
2. Change to the TESTAR_dev directory and execute the following commands
3. ./gradlew init_workflow_test
4. ./gradlew build
5. ./gradlew installDist
6. ./gradlew runTestWebdriverSuspiciousTagMacOS

Chromedriver should execute Chrome to access a web application, detect the existence of GUI widgets, and report the detection of a 'Forum' widget link.

[sureshe456]

Hi @ferpasri We will look into the issue and keep you posted with updates.

[prasanjitsahoo]

Hi @ferpasri ,Could you please run your pipeline once now. I could see below warnings when trying to reproduce the issue. WARNING: Unable to find an exact match for CDP version 130, returning the closest version; found: 129; Please update to a Selenium version that supports CDP version 130

Also, please check selenium version that is compatible with CDP 130 and upgrade it if required. As newer versions may have better compatibility with the newer Chrome DevTools Protocol versions

[ferpasri]

Hi @prasanjitsahoo , Thanks for tracking this issue.

We updated the Chromedriver selenium dependencies to v130, and the issue persists: https://github.com/TESTARtool/TESTAR_dev/actions/runs/11909718690

I have been debugging, and I think I know where the issue comes from:

• TESTAR loads a JavaScript extension into the Chrome browser using the options.addArguments("--load-extension=" + extensionPath); argument
• TESTAR then uses Chromedriver to execute a JavaScript extension method that returns the web state.
• However, this extension does not seem loaded in the browser of the hosted macOS runner image.

I think this is probably due to the browser security options.

[prasanjitsahoo]

HI @ferpasri, Did you get a chance to check the warnings below? BUILD FAILED in 23s Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. The failure occurred at: * Where: Script '/Users/runner/work/TESTAR_dev/TESTAR_dev/testar/workflow.gradle' line: 365.

[ferpasri]

Hi @prasanjitsahoo ,

Do you know if the security options for Chrome on the macOS runner have changed recently?

It seems TESTAR is unable to load a JavaScript extension in the browser. Then, when TESTAR attempts to call the internal JavaScript function that should have been loaded, it fails to retrieve the necessary information.

Regarding the Gradle warning, it indicates we will need to update some Gradle functions when uploading to 9.0 (in the future). Line 365 is the CI pipeline that throws the error because TESTAR could not retrieve the GUI web information to pass the test.

[prasanjitsahoo]

Hi @ferpasri , Thank you for your patience. We are currently investigating the issue and it is in progress. Please allow us some more time to delve deeper.

[prasanjitsahoo]

HI @ferpasri , We have continued investigating this issue and would like to highlight the findings below:

Chrome Drivers Chrome drivers are regularly updated to meet security standards and address vulnerabilities across all images. Apart from version upgrades, no changes have been made to the Chrome Security Policy from our end. The latest versions are as follows:

Google Chrome: 131.0.6778.140 Google Chrome for Testing: 131.0.6778.108 ChromeDriver: 131.0.6778.108

Recommendations

1. Adjust Chrome Security Settings Test by modifying the security settings in Google Chrome, as the issue might stem from blocked insecure content (this is common with tool integrations). Temporarily disable pop-up blockers and Chrome security features for testing purposes. Disable Chrome security flags related to extensions and test again.
2. Screen Recording Conflict If TESTSTAR has screen recording capabilities enabled, disable them and test again to rule out conflicts.
3. Gradle Update Warning Gradle warns that some functions will need to be updated when transitioning to version 9.0. The current latest version is v8.11 (refer to Gradle releases). Review the Gradle documentation for specific instructions on updating these functions to ensure compatibility with Gradle 9.0 and future versions.

[prasanjitsahoo]

As we haven’t received any additional updates from your end, we will proceed to close this issue. If you require further assistance or encounter related challenges, please feel free to reopen the issue or reach out to us.

Thank you for your understanding!