chore: improve fairshare scheduler for heterogeneous agents, improve tests.

[ioga]

Description

• Restructure fairshareScheduler to closer match the priorityScheduler.
• Maintain internal agent state to make it aware of its own scheduling fits to avoid the cases where there're enough slots for multiple tasks, but no fits for all of them. That may happen with heterogeneous agents. See TestFairShare884435Sched88 test case.
• Enable scheduler tests to have tasks scheduled across multiple agents.
• Improve assertion messages for some scheduling checks to make it more evident which particular test or part of a test failed.

Test Plan

No changes should be observed for the standard homogeneous agents setup.

Commentary (optional)

Initially, I've set out to fix a problem I've seen with fairshare scheduling of (2 slot, 2 slot, 1 slot) tasks sequentially on (2 slot, 1 slot) agents that I've seen in December, but when I tried to repro it now or make a test for it, I could not do that, mysteriously. Perhaps something more complex has been happening there, but I can't spend to much time on that. So at least lets improve testing a bit.

Checklist

• [ ] User-facing API changes need the "User-facing API Change" label.
• [ ] Release notes should be added as a separate file under docs/release-notes/. See Release Note for details.
• [ ] Licenses should be included for new code which was copied and/or modified from any external code.

[netlify[bot]]

✔️ Deploy Preview for determined-ui canceled.

🔨 Explore the source changes: 0d0c3faaac4aef565a7c2c4250c34c7ae2db33d5

🔍 Inspect the deploy log: https://app.netlify.com/sites/determined-ui/deploys/61e86dd380f98b0007d9a059

[hamidzr]

I'm sorry don't think I'll find time to review in more detail before my OOO, (i should try to wrap up something else). lmk if there is a section you think I need to see. (or if I should make time)

[ioga]

to be revisited.