feat+fix: preserve test collection order
We run into issues with test ordering with some frequency in the ATS step in CI. This is because ATS scrambles the test order and this uncovers certain dependencies within tests.
Sure there's some value in doing that, but fixing these issues is somewhat annoying and actually disrupts the workflow more than is helpful. Scrambling test order needs to be deliberate, not accidental
These changes make sure that we preserve the order that pytest collected tests when deciding the order to run them.
Notice that it can still run into ordering issues if a test that depends on another test does't get picked to run.
Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:
Project coverage is 95.47%. Comparing base (
4274ae5) to head (a4bad6b). Report is 134 commits behind head on main.
Additional details and impacted files
@@ Coverage Diff @@
## main #346 +/- ##
==========================================
- Coverage 95.52% 95.47% -0.05%
==========================================
Files 80 80
Lines 2746 2764 +18
==========================================
+ Hits 2623 2639 +16
- Misses 123 125 +2
| Flag | Coverage Δ | |
|---|---|---|
| python3.10 | 95.71% <100.00%> (-0.05%) |
:arrow_down: |
| python3.11 | 95.71% <100.00%> (-0.05%) |
:arrow_down: |
| python3.8 | 95.71% <100.00%> (-0.05%) |
:arrow_down: |
| python3.9 | 95.71% <100.00%> (-0.05%) |
:arrow_down: |
| smart-labels | 95.47% <100.00%> (-0.05%) |
:arrow_down: |
Flags with carried forward coverage won't be shown. Click here to find out more.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
@ai-review-prompt-prod testing AI code review, you can ignore the feedback (if any).
@giovanni-guidini is this still needed?
@thomasrockhu-codecov I think it's pretty safe to close this. We no longer need it.