feat: introduce cypress e2e tests

[perzeuss]

Checklist:

• [x] Please open an issue before creating a PR or link to an existing issue
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Description

As described in #9253, this PR introduces cypress e2e tests to test Dify features. The setup is using gherkin & cucumber to define test cases. This makes it easier to write tests in a more human-readable format and also acts as documentation for the features.

Type of Change

• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update, included: Dify Document
• [x] Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• [ ] Dependency upgrade

Testing Instructions

Run docker test environment an execute Cypress tests

• [ ] Go to the /web directory and run yarn to install the dependencies
• [ ] Run yarn test:e2e:ci to run the cypress tests
  • This will run a Dify API and required databases configured in docker/docker-compose.cypress.yaml. Data is stored in a separate volume
  • This will start the frontend devserver
  • Cypress tests are only executed after both the api and frontend are up and running

The tests should pass and you should see the recorded videos in web/cypress/videos. Since the tests run fast, the videos will only be helpful if a test fails. If you want to generate videos that can be used as reference for docs, run yarn test:e2e:docs which will generate tests that are slower and allow to follow to what happens in the frontend.

Opening the Cypress UI

For the development of tests, you would want to open the Cypress UI.

• [ ] Run yarn cy:run, an electron window should open where you can select the e2e test suite and run individual tests
• [ ] If your dev environment runs on a remote machine without GUI, you can visit http://localhost:6080/ to see the GUI via VNC. If you are on windows with wsl, you can run yarn cy:open:wsl and use VcXsrv to connect to the VNC server and interact with the Cypress UI.

GitHub Action

I've added a GitHub action to run the tests. For a performance boost, it will create and use a docker image as cache. For this we need to configure the secret CYPRESS_CONTAINER_REGISTRY_TOKEN (classic token with registry:write permission for docker push) and the variables DOCKER_NAMESPACE and DEVCONTAINER_IMAGE_NAME in the repository settings.

You can find a test run of the GitHub action here: https://github.com/perzeuss/dify/actions/runs/11428886478