usaspending-api icon indicating copy to clipboard operation
usaspending-api copied to clipboard

[DEV-8395] Travis CI based on our docker-compose.yaml services

Open kwhickey opened this issue 3 years ago • 1 comments

Description

Make sure Travis CI workloads and our local dev env workloads (and tests in each) are all using the same config and setup so they pass and fail the same.

Technical Details Background

Travis CI allows “services” to be added to a linux box on which the build will take place. Postgres is one of those. While convenient, it is different than the environment used to develop and run unit tests locally, and the specifics of that postgres environment (like DB version) can drift form local env or from production.

Tech Approach

  1. Get the usaspending-ci container working and doing things that are done in Travis CI (see blocking PR: #3350)
  2. Remove the postgres service from .travis.yml
  3. Remove the elasticsearch service from .travis.yml
  4. Run usaspending-ci in .travis.yml
  5. Backfill env vars for docker-compose run of services
  6. Run and see if test still pass
  7. Compare timing to see if things got much slower

Requirements for PR merge:

  1. [x] Unit & integration tests updated
  2. [NA] API documentation updated
  3. [ ] Necessary PR reviewers:
    • [ ] Backend
    • [NA] Frontend <OPTIONAL>
    • [ ] Operations <OPTIONAL>
    • [NA] Domain Expert <OPTIONAL>
  4. [NA] Matview impact assessment completed
  5. [NA] Frontend impact assessment completed
  6. [NA] Data validation completed
  7. [NA] Appropriate Operations ticket(s) created
  8. [x] Jira Ticket DEV-8395:
    • [x] Link to this Pull-Request
    • [x] Performance evaluation of affected (API | Script | Download)
    • [x] Before / After data comparison

Area for explaining above N/A when needed:

kwhickey avatar Jan 28 '22 15:01 kwhickey

This PR works, but is on-hold. Why: With the current Dockerfile, it adds 5-7min to the CI build. If we change it to be based on a python:3.7 base docker image, that 5-7m is reclaimed. But we need to make sure there are no issues for ops/CI/jobs for rebasing our image on a different one.

kwhickey avatar Mar 07 '22 14:03 kwhickey

Closing as this is probably overcome by events and may not be worth it, given the way travis builds have been parallelized in #3920 . This branch mod/dev-8395-ci-use-docker-postgres should be kept open as an example of how we could 100% dockerize our CI build if we want to in the future.

kwhickey avatar Sep 21 '23 16:09 kwhickey