Add a bisect tool using postsubmit pkgci artifacts
When diagnosing performance regressions it can be useful to run a bisect process to identify culprit commits. Git has a built in bisect tool: https://git-scm.com/docs/git-bisect, but bisecting across commits requires source builds that can be slow (particularly across LLVM commits). A full source build bisect with <10 steps can take a few hours if each build is 10-20 minutes long.
Postsubmit CI runs produce artifacts like the linux_x86_64_release_packages (68MB) on https://github.com/openxla/iree/actions/runs/7978888274. We could bisect across commit ranges by downloading those release packages and running them. We could write a script that uses the GitHub CLI (https://cli.github.com/) to enumerate commits and postsubmit runs, download artifacts, and run a bisect test command (or allow manual marking of good/bad commits).
Nightly releases are also an option, especially when wanting to run on operating systems that pkgci does not currently produce artifacts for, but postsubmit CI runs should have artifacts for every commit with no gaps.
Nice - this is basically the whole reason I set it up this way
Partial approach here: https://github.com/openxla/iree/tree/main/experimental/regression_suite#setting-up-a-venv
You may want to look into elfshaker too: https://github.com/elfshaker/elfshaker. We use this internally for quickly bisecting clang and GCC builds, and I believe it could work for IREE too.
For exampe, manyclangs (https://github.com/elfshaker/manyclangs) lets you download a few fairly small pack files (~few 100MB), then bisect through years of commits way quicker than you'd imagine using elfshaker + a small git bisect script. I've bisected every commit between LLVM 14 to 15 in a few minutes with manyclangs.