Assert in Notify triggered via `JoinHandle.join()`

[bdonlan]

CI on this branch on tokio https://github.com/tokio-rs/tokio/pull/2914 is triggering an assert deep in loom's Notify::wait() function. It can be reliably reproduced by running on that branch:

 (cd tokio; RUST_BACKTRACE=1 RUSTFLAGS='--cfg loom --cfg tokio_unstable -Dwarnings' LOOM_MAX_PREEMPTIONS=2 cargo test --lib --release --features full -- --nocapture pool_shutdown)

Based on initial investigation, it seems that branch_thread selects a non-runnable thread for execution, possibly because the scheduler's execution path is inconsistent with the actual state of the threads in question.

[bdonlan]

Found the cause. There was nondeterminism, in the form of the JoinHandles coming out of a HashMap (where keys are of indeterminate order).