aesara icon indicating copy to clipboard operation
aesara copied to clipboard

Add basic sub-package-level test ordering

Open brandonwillard opened this issue 3 years ago • 2 comments

This PR adds simple sub-package-based test ordering, so that—for instance—tests in tests.graph are run first, then tests.compile, etc.

Since pytest-order doesn't appear to have a straightforward way to order tests based on directories/sub-packages (at least not without adding marks to every module in a sub-package), the approach taken here doesn't use it; however, we definitely want the granularity offered by pytest-order, because it would be better if we could order tests within and across sub-packages and modules. Nevertheless, these changes are minimal and at least an improvement over the current order-less approach, so we can easily revert, follow-up, and/or augment them with pytest-order in the near future.

Closes #730

brandonwillard avatar Mar 10 '22 23:03 brandonwillard

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 78.44%. Comparing base (3d8553f) to head (8640697). :warning: Report is 727 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #853   +/-   ##
=======================================
  Coverage   78.44%   78.44%           
=======================================
  Files         154      154           
  Lines       47749    47749           
  Branches    10848    10848           
=======================================
  Hits        37456    37456           
  Misses       7750     7750           
  Partials     2543     2543           
:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar Mar 11 '22 00:03 codecov[bot]

It doesn't look like the ordering is taking effect in CI, but it does locally.

brandonwillard avatar Mar 11 '22 18:03 brandonwillard