Scott Todd

## Existing test suites This page documents many of the tests that we have in-tree and out-of-tree: https://iree.dev/developers/general/testing-guide/. Notably missing there are explanations for [`tests/e2e/matmul`](https://github.com/iree-org/iree/tree/main/tests/e2e/matmul) and [`tests/e2e/convolution`](https://github.com/iree-org/iree/tree/main/tests/e2e/convolution), which the draft...

RE: performance tests / benchmarks, I think an offline generation approach helps manage complexity there too: * A Python script generates the sweep across dimensions along with reference outputs *...

For just testing, 1 week seems like a reasonable target. I wouldn't want to retrofit benchmarking onto the `iree_generated_e2e_runner_test` code, at least staying in the context of CMake or Bazel....

Pkgci is just a set of actions that 1. build the python packages and install them 2. run tests with the installed packages The older ci.yml workflows rely on a...

> Doesn't pkgCI also use the built iree-compile/benchmark-module by somwhat pointing to the build path Those tools are available as console scripts after installing python packages: https://github.com/iree-org/iree/blob/3dffadbc0b8a37ab170a61eddb3131c8cbd8c2b2/compiler/setup.py#L454-L463 https://github.com/iree-org/iree/blob/3dffadbc0b8a37ab170a61eddb3131c8cbd8c2b2/runtime/setup.py#L599-L609 Pkgci is...

> Also do you have thoughts/ideas on best way to build out the attention perf benchmarks would you recommend any one of these existing frameworks `(longitudinal on perf.iree.dev, comparative with...

> Also if you want you can fork the repo https://github.com/iree-gd/iree.zoo/ and run convert action from there to see the logs (Not directly related to this issue) This is cool!...

> I could make either a master issue for all things that fail, also I can make a big list of all modules might be good to use (was thinking...

Can we alias the flags? Changing the flag names once caused some disruption.

> Our downstream project IREE does run good torch node tests, but [reports many of the onnx ops that we've lowered as failing](https://github.com/iree-org/iree/blob/main/build_tools/pkgci/external_test_suite/onnx_gpu_vulkan.json). I haven't found a way to view...