Scott Todd

I also tried running the https://github.com/onnx/onnx/tree/main/onnx/backend/test/data/simple tests, which are a step higher level than the "node" tests. 2/23 of those passed, but I don't think the models themselves there are...

I have an initial sketch of a new test suite and supporting scripts up at https://github.com/nod-ai/SHARK-TestSuite/pull/68 . That will go through a few iterations of refinement then I'd like to...

More progress 😄! I have a GitHub Actions workflow and rough sketch of pytest support now. Stacking changes at https://github.com/ScottTodd/SHARK-TestSuite/tree/iree-pytest for now. Here's a sample workflow run with a known...

Got a simple config file (written in JSON for now, can reach for something more complex as needed): https://gist.github.com/ScottTodd/b93c81aa95fdc1d644596b931ceac05e Then ran with https://pypi.org/project/pytest-xdist/: ``` D:\dev\projects\SHARK-TestSuite (iree-pytest) (.venv) λ pytest iree_tests...

Added `iree-run-module` tests and a whole bunch of quality of life tweaks to the pytest configuration. Now works on GitHub Actions too: https://github.com/ScottTodd/SHARK-TestSuite/actions/runs/8056608699/job/22006010154 ``` ============================= test session starts ============================== platform...

The StableHLO tests from here: https://github.com/openxla/stablehlo/tree/main/stablehlo/tests/interpret could be used if we wrote a converter script / (MLIR pass?) to replace the `check` ops with inputs and expected outputs (or just...

As these new tests prove themselves, I'm actually keeping an eye on the check dialect in IREE (https://iree.dev/reference/mlir-dialects/Check/) and if it and the associated infrastructure are worth keeping around. We've...

Initial work has landed and tests are running. Spotted one apparent flake already: https://github.com/openxla/iree/actions/runs/8086418421/job/22096503710?pr=16603#step:9:41 (compiler crash on `FAILED SHARK-TestSuite/iree_tests/onnx/node/generated/test_layer_normalization_2d_axis1/model.mlir::gpu_rocm_rdna3`)

Current status: * Most ONNX ops were imported successfully from the upstream test suite. There are 48 import failures: https://github.com/nod-ai/SHARK-TestSuite/blob/main/iree_tests/onnx/node/import_failures.txt (a mix of sequence types, optional types, and a few...

Going to call this fixed now. We're continuing to add * test cases under https://github.com/nod-ai/SHARK-TestSuite/tree/main/iree_tests * configs at https://github.com/openxla/iree/tree/main/build_tools/pkgci/external_test_suite * CI jobs at e.g. https://github.com/openxla/iree/blob/main/.github/workflows/pkgci_regression_test_cpu.yml