Running benchmarks is blocked by build_all

[hanhanW]

I recently noticed that running benchmarks is blocked by build_all job after https://github.com/openxla/iree/commit/f1adc7897c620fd835da718c68ce0c4156d45688. This is not convenience because we could just test performance before fixing all the lit tests. Disabling failing tests seems not help. My workaround is deleting the tests in CI jobs. I.e., delete the below lines:

https://github.com/openxla/iree/blob/d42e457d0a92a79b309715317c50c8012b073a4a/.github/workflows/build_all.yml#L79-L90

Sample log: https://github.com/openxla/iree/actions/runs/8575069562?pr=16456

@ScottTodd is there a plan to move it out of build_all? It is very helpful if we can run benchmarks even when there are lit tests failing.

[ScottTodd]

The long term fix is to refactor the benchmark jobs to only depend on release artifacts. That would currently mean anchoring on the newer "pkgci" jobs, or forking off into a dedicated "benchmark" set of jobs that follow the same design.

https://github.com/openxla/iree/issues/16203 and https://github.com/openxla/iree/issues/16431 started moving in that direction, but they did move some test actions into the existing build_all job, yes. We previously had build_all -> test_all with the full build directory uploaded and then downloaded between jobs, but that work switched to only transferring the "install" directory.

We might be able to split build_all back into build -> test_cpu but I'd need to check what test coverage we could get that way. I'd rather run CPU unit tests (compiler lit tests) with the full build directory already on disk...

[ScottTodd]

I think we should move the benchmark jobs moved out of ci.yml (I initially pushed for them to not start there in the first place... but that's the current reality). That will duplicate some CI time on a CPU build machine but then benchmarking will be decoupled from unit tests.

[ScottTodd]

Pushed two test PRs with different approaches. Seeing if either works and is worth pursuing further.

1. Split ci.yml into ci.yml and benchmark.yml (probably good long term, but duplicates some work will need further refactoring)
2. Try to fail ci.yml if tests in build_all fail but still run benchmarks if the build succeeded (hacky, difficult to reason about)

[ScottTodd]

I think this is solved well enough for now by https://github.com/iree-org/iree/pull/17400. The build system setup and github actions workflows could use more refactoring to support packages instead of the install dir though.

[hanhanW]

Yes, this is solved well enough, thanks Scott!