Improve Evaluation Performance

[baronfel]

• [ ] Eric Arndt's Eval Perf PRs
  • I've asked Eric to help us with splitting #11273 to smaller pieces.
    • [x] #11780
    • [x] #11779
    • We landed most of Erics PRs, two remain.
    • [ ] #11852 - looks good, for some reason two tests failed. But some of our tests are flaky - re running.
    • [ ] #11802 - killed pipelines in a weird way, hanging since. @Erarndt
  • #11271 - Approved. Waiting for VS insertion to finish. There were some issues with tests, possibly unrelated. Re-running the pipeline.
• [ ] Eric Arndt spike multi-threaded eval
  • discussion in progress. Will update as we go along and eventually create a separate ticket.
  • We're moving ahead with this one. There will be a new ticket soon-ish I think.