sahat
test(e2e): add End-to-End (E2E) tests for the RAG Integration Example
Update Oct 22: As we are approaching the end of October, I'll accept the first PR that is ready to be merged. No need for an exclusive issue assignment. Make sure to go over this entire issue description including its notes.
Prerequisites:
- You must have obtained a valid Together.ai API Key and a Hugging Face API Key - see https://github.com/sahat/hackathon-starter?tab=readme-ov-file#obtaining-api-keys
- Have Hackathon Starter up and running on MongoDB Atlas with the above key and be able to load /ai/rag , try it, and understand what it does.
- Be able to run
npm run test:e2e-nokeywith all existing tests passing.
Scope:
- Create
test/e2e/rag.e2e.test.jsfor E2E testing of /ai/rag with Playwright. Seetest/e2e-nokey/github-api.e2e.test.jsas an example. - The tests must cover all data paths.
npx playwright test --config=test/playwright.config.js test/e2e/rag.e2e.test.jsshould run the existing tests and your new tests, and all should be passing.
Please include a screenshot of the test report with your submission.
Note 1: You would need to complete the prerequisites before asking for GitHub issue assignment. Note 2: Please don't just copy-paste AI-generated content (code, issue replies, or PR text), as such submissions will be rejected and may be marked as spam. You can use AI assistants to be more efficient, but the final result should be your work with the help of an assistant, not an assistant's output taken at face value and copy-pasted.
@YasharF I have set up all the prerequisites, please assign me this as part to hacktober fest 2025.
@YasharF I plan to cover the following cases for /ai/rag:
- Page loads with all UI elements visible.
- Valid query returns a proper API response.
- Empty input shows validation/error message.
- Edge inputs (long/special chars) handled gracefully.
- Simulated API errors display proper error messages.
- Loading spinner appears/disappears correctly.
~~Why those?~~
There is a big mismatch between what you are planning and what we need. I am going keep this with help wanted label for someone else to pick up.
Hello @YasharF,
I would like to work on this issue for Hacktoberfest. I have fulfilled the prerequisites and have read the previous conversation. I understand the goal is to test the RAG data paths, not just the UI.
My proposed test plan focuses on this:
- Successful RAG Path: Programmatically ingest a known document, ask a question that requires its content, and assert that the final LLM response is correct.
- Out-of-Context Path: Use the same document but ask an unrelated question, asserting that the system gracefully states it doesn't know the answer.
This approach will validate the full data flow: ingestion, embedding, retrieval, and generation. Please let me know if this aligns with what you're looking for.
- You would also need to consider the database state, and how you will need to reset the state between tests. The database state changes take some time, so you would also need to wait properly and check for the state updates. The ap.js file has code that does database and vector search initialization code that you can look at to get an idea of how to setup the tests to do the same for state resets.
- The other item to consider is the cache. You would need to have calls that tests both cached paths and non-cached paths, which means you would need to vacate the cache in certain cases.
Considering these, you are probably going to end up with more test cases. Let me know if you have questions and if you are comfortable with dealing all the different caches and database states. If you are good let me know about your updated test cases.
Reminder to all, as stated in the Issue description:
Note 2: Please don't just copy-paste AI-generated content (code, issue replies, or PR text), as such submissions will be rejected and may be marked as spam.
