codecov-cli icon indicating copy to clipboard operation
codecov-cli copied to clipboard

feat+fix: preserve test collection order

Open giovanni-guidini opened this issue 1 year ago • 2 comments

We run into issues with test ordering with some frequency in the ATS step in CI. This is because ATS scrambles the test order and this uncovers certain dependencies within tests.

Sure there's some value in doing that, but fixing these issues is somewhat annoying and actually disrupts the workflow more than is helpful. Scrambling test order needs to be deliberate, not accidental

These changes make sure that we preserve the order that pytest collected tests when deciding the order to run them.

Notice that it can still run into ordering issues if a test that depends on another test does't get picked to run.

giovanni-guidini avatar Jan 02 '24 20:01 giovanni-guidini

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 95.47%. Comparing base (4274ae5) to head (a4bad6b). Report is 134 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #346      +/-   ##
==========================================
- Coverage   95.52%   95.47%   -0.05%     
==========================================
  Files          80       80              
  Lines        2746     2764      +18     
==========================================
+ Hits         2623     2639      +16     
- Misses        123      125       +2     
Flag Coverage Δ
python3.10 95.71% <100.00%> (-0.05%) :arrow_down:
python3.11 95.71% <100.00%> (-0.05%) :arrow_down:
python3.8 95.71% <100.00%> (-0.05%) :arrow_down:
python3.9 95.71% <100.00%> (-0.05%) :arrow_down:
smart-labels 95.47% <100.00%> (-0.05%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Jan 02 '24 20:01 codecov[bot]

@ai-review-prompt-prod testing AI code review, you can ignore the feedback (if any).

eliatcodecov avatar Jul 08 '24 14:07 eliatcodecov

@giovanni-guidini is this still needed?

thomasrockhu-codecov avatar Feb 11 '25 15:02 thomasrockhu-codecov

@thomasrockhu-codecov I think it's pretty safe to close this. We no longer need it.

giovanni-guidini avatar Feb 11 '25 16:02 giovanni-guidini