restricted-site-access icon indicating copy to clipboard operation
restricted-site-access copied to clipboard

Inconsistent test results between local and Github Actions environment

Open Sidsector9 opened this issue 2 years ago • 2 comments

Describe the bug

Tests pass locally but fail on GitHub actions. So far the cause is unknown.

  • Tests pass for WP:latest but fail for WP:trunk and WP:mininmum
  • This issue was first detected in PR #169 at specifically this line.
  • This is the GitHub Actions log pointing to the test failure.
  • The only way to make the tests pass was to rewrite the test with an alternative approach.

What has been done so far to debug this?

With @dinhtungdu's suggestion, we debugged every test case with the help of Cypress's screenshot and video recording features, which were uploaded using GitHub Actions artifacts.

This is the video artifact that shows the failure on Github Actions:

https://user-images.githubusercontent.com/17757960/145206299-def50e06-3904-4c2c-93ee-317e477c2838.mp4

It is clear from the video that redirecting to /accessible-page returns a 404 despite the page being published.

Steps to Reproduce

  1. Fork this repo
  2. After setting up the local environment, modify the test cases to the failing state as seen in PR #169
  3. Run npm run wp-env start && npm run create-pages
  4. Run npm run cypress:run
  5. See that all test cases pass
  6. Commit and push and check GitHub actions, 1 test fails

Expected behavior Cypress behavior should be consistent between local and CI environments.

Sidsector9 avatar Dec 08 '21 12:12 Sidsector9

@Sidsector9 did #171 resolve this?

jeffpaul avatar May 23 '22 17:05 jeffpaul

Jeff PR #171 seem to be unrelated to this issue. @mitogh can you please confirm?

Sidsector9 avatar Jun 08 '22 11:06 Sidsector9

Closing this as it is an edge case. Will reopen if the issue reoccurs.

Sidsector9 avatar Oct 15 '22 05:10 Sidsector9