Scott Todd

FWIW, I looked at [the logs for a sample run of `build_e2e_test_artifacts`](https://github.com/openxla/iree/actions/runs/7923298643/job/21633120644) to see if there were some easy optimizations to make and it doesn't look like it. * 4m...

I'm seeing if I can pull some statistics for how long each job spends queued right now. (If that ends up taking longer than I'd like I may just proceed...

Slight adjustments to the plans from the initial issue description: * `build_benchmark_tools`, `build_e2e_test_artifacts`, and `test_benchmark_suites` are now opt-in and will only run when benchmarks are requested (including all postsubmit runs)...

Well, https://github.com/iree-org/iree-samples/actions/workflows/regression_tests.yml / https://github.com/iree-org/iree-samples/blob/main/.github/workflows/regression_tests.yml (out-of-tree, unmonitored) is probably not a great reference - that's been failing for 7 months without much attention.

I started looking through https://github.com/nod-ai/SHARK-TestSuite/tree/main/e2eshark in more detail today. * I'm not sure how much reusable from https://github.com/nod-ai/SHARK-TestSuite/blob/main/e2eshark/run.py or the setup instructions in the README. That's all pretty bespoke and...

[`experimental/regression_suite`](https://github.com/openxla/iree/tree/main/experimental/regression_suite) has approximately the shape of what I'd like to see in a test runner + suite configuration. A few things potentially missing * [`build_tools/pkgci/setup_venv.py`](https://github.com/openxla/iree/blob/main/build_tools/pkgci/setup_venv.py) and pkgci is currently only...

Seems like pkgci is running 'presubmit' tests on presubmit _and_ postsubmit, and never runs 'postsubmit' tests? https://github.com/openxla/iree/blob/c02b89e3c7e22eff009fc318132b5ed3fe9a2d97/.github/workflows/pkgci_regression_test_cpu.yml#L51-L56

The ONNX test suite looks promising: * https://onnx.ai/onnx/repo-docs/ImplementingAnOnnxBackend.html * https://onnx.ai/onnx/repo-docs/OnnxBackendTest.html * (sources) https://github.com/onnx/onnx/blob/main/onnx/backend/test/case/node/matmul.py * (generated) https://github.com/onnx/onnx/tree/main/onnx/backend/test/data/node * (generated) https://github.com/onnx/onnx/tree/main/onnx/backend/test/data/node/test_matmul_2d That style of having Python scripts generate `model.onnx` and input/output protobuf...

I've made progress on this using this script: https://github.com/ScottTodd/iree/blob/tests-regression-onnx/experimental/regression_suite/scripts/run_onnx_tests.py That uses a checkout of the ONNX repo to convert their generated test cases from this format: ``` onnx/backend/test/data/node/... test_foo/ model.onnx...

> - [ ] Push the generated .mlir, .npy, and .txt files to a git repo (possibly with [LFS](https://git-lfs.com/)) Regarding this, I'm also toying with the idea of taking a...