HomeUniteUs
HomeUniteUs copied to clipboard
675 Add E2E Tests to CI without Mocking
Closes #675.
What changes did you make?
- Updated our CI pipeline to run our full e2e test suite against our real backend. This was most easily achieved using our docker container, since GitHub actions are able to manage container lifecycles when run using service containers.
- Refactored the existing pipeline steps to use our docker images. This adds some automated build checks for our images, and simplifies the CI pipeline setup.
- Add a health check endpoint to the API, and use it to verify that the API container startup was successful.
- Fix minor bug in
app/cypress/e2e/create-new-password.cy.ts
. The test was short-circuiting the hostname config options. - bump
cryptography
to42.0.4
since41.0.7
has a known security vulnerability-
cryptography
is a transitive dependency of themoto
library. Unfortunately the most recent versions ofmoto
still use the outdated version so I had to elevatecrypography
to ourpyproject.toml
file.
-
It is worth noting that our API container is being used as an API and a database. In the future we might consider breaking out the database into a separate container, and populating it with some test data. I'm leaving as-is for now since our db is relatively simple at this point.
Rationale behind the changes?
- Our real e2e tests have broken a few times because they are a pain to run locally. This CI pipeline will prevent PR merges that break our e2e tests.
Testing done for these changes
- My biggest concern was verifying that the
test-api-real-aws
uses our real AWS service, and thattest-app-real-backend
uses the real backend. I verified both in d2e8836469f5b490ea4029951a9abd6210fd9cec by removing the AWS secret from the test config and remove the backend service from the CI pipeline. This change broke the "nomock" test suites, while the "mock" test suites continued to pass.- Also, the runtimes for the nomock test suites are significantly longer
What did you learn or can share that is new?(optional)
- GitHub service containers make it very easy to orchestrate multiple services
- The docker HEALTHCHECK command is very useful for running service containers with a lengthy startup process. GitHub action service containers will query the state of service containers using
docker inspect
. It does this query to determine when the container is ready for use. If there is no HEALTHCHECK thendocker inspect
returns a0
exit code as soon as the container is initialized. A custom health check will ensure thatdocker inspect
does not return0
until the condition you define is met. In our case, the GH action should not progress until the API is actually available.
Screenshots of Proposed Change
Here is the new CI pipeline flow:
I'm converting this to draft. I have some ideas on how to speed up the CI pipeline, but I'm going to implement #636 over these improvements.