haystack icon indicating copy to clipboard operation
haystack copied to clipboard

Add End-to-End Tests

Open julian-risch opened this issue 2 years ago • 0 comments

We run unit tests and integration tests for Haystack but don't have end-to-end tests at the moment. Nightly tutorial runs extend the current testing a bit.

After adding benchmarks in Q2, we should now introduce end-to-end tests that can be run nightly or at least for release candidates. These end-to-end tests can include longer running tests and should mimic real use cases of Haystack, for example, full pipelines, with indexing and querying. End-to-end tests will also allow to spot issues with the pipeline v2 migration early on.

One starting point can be the outdated https://github.com/deepset-ai/haystack/blob/main/.github/workflows/e2e.yml

Core engineering efforts

  • Define usage scenarios to cover with end-to-end tests (use feedback from other teams)
  • Implement set of initial end-to-end tests for Haystack 1.x and 2.x
  • Set up workflow that runs end-to-end tests (manually triggered)

Developer relations efforts

  • Help defining usage scenarios to cover with end-to-end tests
### Tasks
- [ ] https://github.com/deepset-ai/haystack/issues/5333
- [ ] https://github.com/deepset-ai/haystack/issues/5350
- [ ] https://github.com/deepset-ai/haystack/issues/5335
- [ ] https://github.com/deepset-ai/haystack/issues/5336
- [ ] https://github.com/deepset-ai/haystack/issues/5337
- [ ] https://github.com/deepset-ai/haystack/issues/5334
- [ ] https://github.com/deepset-ai/haystack/issues/5338

julian-risch avatar Jul 04 '23 11:07 julian-risch