haystack
haystack copied to clipboard
Add End-to-End Tests
We run unit tests and integration tests for Haystack but don't have end-to-end tests at the moment. Nightly tutorial runs extend the current testing a bit.
After adding benchmarks in Q2, we should now introduce end-to-end tests that can be run nightly or at least for release candidates. These end-to-end tests can include longer running tests and should mimic real use cases of Haystack, for example, full pipelines, with indexing and querying. End-to-end tests will also allow to spot issues with the pipeline v2 migration early on.
One starting point can be the outdated https://github.com/deepset-ai/haystack/blob/main/.github/workflows/e2e.yml
Core engineering efforts
- Define usage scenarios to cover with end-to-end tests (use feedback from other teams)
- Implement set of initial end-to-end tests for Haystack 1.x and 2.x
- Set up workflow that runs end-to-end tests (manually triggered)
Developer relations efforts
- Help defining usage scenarios to cover with end-to-end tests
### Tasks
- [ ] https://github.com/deepset-ai/haystack/issues/5333
- [ ] https://github.com/deepset-ai/haystack/issues/5350
- [ ] https://github.com/deepset-ai/haystack/issues/5335
- [ ] https://github.com/deepset-ai/haystack/issues/5336
- [ ] https://github.com/deepset-ai/haystack/issues/5337
- [ ] https://github.com/deepset-ai/haystack/issues/5334
- [ ] https://github.com/deepset-ai/haystack/issues/5338