HomeUniteUs icon indicating copy to clipboard operation
HomeUniteUs copied to clipboard

675 Add E2E Tests to CI without Mocking

Open Joshua-Douglas opened this issue 10 months ago • 1 comments

Closes #675.

What changes did you make?

  • Updated our CI pipeline to run our full e2e test suite against our real backend. This was most easily achieved using our docker container, since GitHub actions are able to manage container lifecycles when run using service containers.
  • Refactored the existing pipeline steps to use our docker images. This adds some automated build checks for our images, and simplifies the CI pipeline setup.
  • Add a health check endpoint to the API, and use it to verify that the API container startup was successful.
  • Fix minor bug in app/cypress/e2e/create-new-password.cy.ts. The test was short-circuiting the hostname config options.
  • bump cryptography to 42.0.4 since 41.0.7 has a known security vulnerability
    • cryptography is a transitive dependency of the moto library. Unfortunately the most recent versions of moto still use the outdated version so I had to elevate crypography to our pyproject.toml file.

It is worth noting that our API container is being used as an API and a database. In the future we might consider breaking out the database into a separate container, and populating it with some test data. I'm leaving as-is for now since our db is relatively simple at this point.

Rationale behind the changes?

  • Our real e2e tests have broken a few times because they are a pain to run locally. This CI pipeline will prevent PR merges that break our e2e tests.

Testing done for these changes

  • My biggest concern was verifying that the test-api-real-aws uses our real AWS service, and that test-app-real-backend uses the real backend. I verified both in d2e8836469f5b490ea4029951a9abd6210fd9cec by removing the AWS secret from the test config and remove the backend service from the CI pipeline. This change broke the "nomock" test suites, while the "mock" test suites continued to pass.
    • Also, the runtimes for the nomock test suites are significantly longer

What did you learn or can share that is new?(optional)

  • GitHub service containers make it very easy to orchestrate multiple services
  • The docker HEALTHCHECK command is very useful for running service containers with a lengthy startup process. GitHub action service containers will query the state of service containers using docker inspect. It does this query to determine when the container is ready for use. If there is no HEALTHCHECK then docker inspect returns a 0 exit code as soon as the container is initialized. A custom health check will ensure that docker inspect does not return 0 until the condition you define is met. In our case, the GH action should not progress until the API is actually available.

Screenshots of Proposed Change

Here is the new CI pipeline flow:

image

Joshua-Douglas avatar Apr 16 '24 04:04 Joshua-Douglas

I'm converting this to draft. I have some ideas on how to speed up the CI pipeline, but I'm going to implement #636 over these improvements.

Joshua-Douglas avatar Apr 29 '24 03:04 Joshua-Douglas