fides icon indicating copy to clipboard operation
fides copied to clipboard

parallelize api and non-api `ops-unit` tests

Open adamsachs opened this issue 4 months ago • 4 comments

Description Of Changes

Splits out execution of ops unit tests under tests/ops/api from other ops unit tests, so they can run in parallel in CI.

This felt like low-hanging fruit to shave off some of our CI runtime for backend checks. BE checks took a total of 25 minutes with these changes, as opposed to around ~35 minutes without it, (e.g. https://github.com/ethyca/fides/actions/runs/7803890822).

As of now, it seems like our /api and non-/api unit tests have similar total runtimes. Unclear whether that will stay the same over time; if it becomes skewed, this parallelization will be less effective and we'd want to switch approaches. In any case, this is not a comprehensive solution to relatively slow test runtimes -- we can and should continue to explore more ways to improve our test runtimes. Namely, this particular change won't have any impact on test runtimes when running isolated/targeted tests locally.

Code Changes

  • [x] support pytest(ops-unit-api) and pytest(ops-unit-non-api) nox sessions, which allow for parallelization of test execution in CI

Steps to Confirm

  • [x] CI still runs to completion, significantly faster. we still have all the same test coverage.

Pre-Merge Checklist

adamsachs avatar Feb 06 '24 17:02 adamsachs

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
fides-plus-nightly ✅ Ready (Inspect) Visit Preview 💬 Add feedback Mar 21, 2024 5:19pm

vercel[bot] avatar Feb 06 '24 17:02 vercel[bot]

Passing run #6809 ↗︎

0 4 0 0 Flakiness 0
⚠️ You've recorded test results over your free plan limit.
Upgrade your plan to view test results.

Details:

Merge 224380296b033d12dccdca093f9a4dfdd47d4bc2 into d0988077413d957a432c22735578...
Project: fides Commit: d8a181edd9 ℹ️
Status: Passed Duration: 00:37 💡
Started: Mar 21, 2024 5:27 PM Ended: Mar 21, 2024 5:28 PM

Review all test suite changes for PR #4595 ↗︎

cypress[bot] avatar Feb 06 '24 18:02 cypress[bot]

this could probably use a bit of polish, but wanted to at least first make sure no one saw any red flags with this incremental update!

adamsachs avatar Feb 06 '24 20:02 adamsachs

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 86.64%. Comparing base (d098807) to head (2243802).

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #4595       +/-   ##
===========================================
+ Coverage   66.33%   86.64%   +20.30%     
===========================================
  Files         338      338               
  Lines       19993    19993               
  Branches     2555     2555               
===========================================
+ Hits        13263    17323     +4060     
+ Misses       6257     2201     -4056     
+ Partials      473      469        -4     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Feb 06 '24 20:02 codecov[bot]