iree icon indicating copy to clipboard operation
iree copied to clipboard

Start cleaning up tests/e2e/.

Open ScottTodd opened this issue 1 year ago • 2 comments

(This will need a few rounds of CI runs to reflect the latest test statuses and fully sync test suites across backends/configs)

Assorted cleanup across the tests/e2e/ folder:

  • Remove some redundant or incorrect comments
  • Moved some Metal and ROCm tests from being exclusively defined in CMake to being defined (but then no-op'd) in Bazel
    • Taught the test function to insert --iree-rocm-target-chip=${IREE_HIP_TEST_TARGET_CHIP} (not happy that this is required though)
  • Added some rocm/hip tests that were missing
  • Merged test srcs down to a single ALL_SRCS glob for test suites that work across all configurations
    • Some of this is wishful thinking and may need to be split back out
    • If these actually work, we can switch from iree_check_single_backend_test_suite to just iree_check_test_suite
  • Added missing tests target collections to test_suite (could also just remove these, they are a convenience for Bazel users)
  • Tried enabling previously disabled tests (some were tagged with issues that were tentatively marked fixed and we don't support XFAIL tests here)
    • Some are known-failing (e.g. attention tests) but we don't have enough visibility into how bad the problems are

ScottTodd avatar Jun 27 '24 22:06 ScottTodd

nice! thanks for looking at this - tests that only run on certain build systems, on certain targets, are persistently disabled, or run only with specific non-standard configuration is a big scaling tax.

benvanik avatar Jun 27 '24 22:06 benvanik

More ideas for refactoring: https://github.com/iree-org/iree/issues/4644#issuecomment-1832433233

(deciding between scoped, incremental improvements and hack-and-slash refactorings...)

ScottTodd avatar Jun 27 '24 23:06 ScottTodd

Closing in favor of smaller PRs.

ScottTodd avatar Jul 10 '24 23:07 ScottTodd