UB in the batch_semaphore linked list

[Darksonn]

The following code has UB:

use futures::future::FutureExt;

fn main() {
    let rt = tokio::runtime::Builder::new_current_thread()
        .build()
        .unwrap();

    let m = tokio::sync::Mutex::new("foo".to_string());

    let lock = rt.block_on(m.lock());
    let l2 = m.lock();
    tokio::pin!(l2);

    (&mut l2).now_or_never();

    drop(lock);
    rt.block_on(m.lock());
}

To see this, run it with:

MIRIFLAGS="-Zmiri-track-raw-pointers" cargo +nightly miri run

The problem was discovered by @kprotty on the discord server.

thread 1:

• poll() creates a Pin<&mut Waiter> with the intrusive node
• it sees theres no permits, grabs the queue lock, and pushes its intrusive node into the queue
• poll() returns thread 1:
• poll() is getting called again (maybe from select!()) and creates a Pin<&mut Waiter>
• thread 1 gets preempted thread 2:
• another thread releases some permits and wants to wake up a Waiter in the queue
• it first grabs the queue lock, then pops thread 1's Waiter node from the queue
• in order to pop the Node, it creates a &Node reference to it to deref some linked-list fields: https://docs.rs/tokio/1.0.1/src/tokio/util/linked_list.rs.html#114

there now exists at the same time:

• thread 1: a Pin<&mut Waiter> which transitively implies a &mut Node in the Waiter
• thread 2: a &Node when trying to pop it

The solution probably looks something like putting the Waiter in Acquire inside an UnsafeCell and only using raw pointers to access it, instead of mutable references.

Click to see miri failure

error: Undefined Behavior: trying to reborrow for SharedReadWrite at alloc1987, but parent tag <14056> does not have an appropriate item in the borrow stack
   --> /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/non_null.rs:212:18
    |
212 |         unsafe { &*self.as_ptr() }
    |                  ^^^^^^^^^^^^^^^ trying to reborrow for SharedReadWrite at alloc1987, but parent tag <14056> does not have an appropriate item in the borrow stack
    |
    = help: this indicates a potential bug in the program: it performed an invalid operation, but the rules it violated are still experimental
    = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
            
    = note: inside `std::ptr::NonNull::<alloc::sync::ArcInner<tokio::runtime::basic_scheduler::Shared>>::as_ref` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ptr/non_null.rs:212:18
    = note: inside `std::sync::Arc::<tokio::runtime::basic_scheduler::Shared>::inner` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/sync.rs:1034:18
    = note: inside `<std::sync::Arc<tokio::runtime::basic_scheduler::Shared> as std::clone::Clone>::clone` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/alloc/src/sync.rs:1295:24
    = note: inside `<std::mem::ManuallyDrop<std::sync::Arc<tokio::runtime::basic_scheduler::Shared>> as std::clone::Clone>::clone` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/mem/manually_drop.rs:50:5
    = note: inside `tokio::util::wake::inc_ref_count::<tokio::runtime::basic_scheduler::Shared>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/util/wake.rs:57:38
    = note: inside `tokio::util::wake::clone_arc_raw::<tokio::runtime::basic_scheduler::Shared>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/util/wake.rs:65:5
    = note: inside `<std::task::Waker as std::clone::Clone>::clone` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/task/wake.rs:268:29
    = note: inside closure at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/sync/batch_semaphore.rs:379:31
    = note: inside `tokio::loom::std::unsafe_cell::UnsafeCell::<std::option::Option<std::task::Waker>>::with_mut::<(), [closure@tokio::sync::batch_semaphore::Semaphore::poll_acquire::{closure#0}]>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/loom/std/unsafe_cell.rs:14:9
    = note: inside `tokio::sync::batch_semaphore::Semaphore::poll_acquire` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/sync/batch_semaphore.rs:370:9
    = note: inside `<tokio::sync::batch_semaphore::Acquire as futures::Future>::poll` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/sync/batch_semaphore.rs:443:15
    = note: inside closure at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/sync/mutex.rs:299:9
    = note: inside `<std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::acquire::{closure#0} for<'r, 's, 't0> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, tokio::sync::Mutex<std::string::String>, &'s tokio::sync::batch_semaphore::Semaphore, tokio::sync::batch_semaphore::Semaphore, u32, tokio::sync::batch_semaphore::Acquire<'t0>, ()}]> as futures::Future>::poll` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:80:19
    = note: inside closure at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/sync/mutex.rs:263:9
    = note: inside `<std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]> as futures::Future>::poll` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/mod.rs:80:19
    = note: inside `<std::pin::Pin<&mut std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>> as futures::Future>::poll` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/future/future.rs:119:9
    = note: inside closure at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/runtime/basic_scheduler.rs:184:58
    = note: inside closure at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/coop.rs:121:9
    = note: inside `std::thread::LocalKey::<std::cell::Cell<tokio::coop::Budget>>::try_with::<[closure@tokio::coop::with_budget<std::task::Poll<tokio::sync::MutexGuard<std::string::String>>, [closure@tokio::runtime::basic_scheduler::Inner<tokio::runtime::driver::Driver>::block_on<std::pin::Pin<&mut std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>>>::{closure#0}::{closure#0}]>::{closure#0}], std::task::Poll<tokio::sync::MutexGuard<std::string::String>>>` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:272:16
    = note: inside `std::thread::LocalKey::<std::cell::Cell<tokio::coop::Budget>>::with::<[closure@tokio::coop::with_budget<std::task::Poll<tokio::sync::MutexGuard<std::string::String>>, [closure@tokio::runtime::basic_scheduler::Inner<tokio::runtime::driver::Driver>::block_on<std::pin::Pin<&mut std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>>>::{closure#0}::{closure#0}]>::{closure#0}], std::task::Poll<tokio::sync::MutexGuard<std::string::String>>>` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/thread/local.rs:248:9
    = note: inside `tokio::coop::with_budget::<std::task::Poll<tokio::sync::MutexGuard<std::string::String>>, [closure@tokio::runtime::basic_scheduler::Inner<tokio::runtime::driver::Driver>::block_on<std::pin::Pin<&mut std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>>>::{closure#0}::{closure#0}]>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/coop.rs:114:5
    = note: inside `tokio::coop::budget::<std::task::Poll<tokio::sync::MutexGuard<std::string::String>>, [closure@tokio::runtime::basic_scheduler::Inner<tokio::runtime::driver::Driver>::block_on<std::pin::Pin<&mut std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>>>::{closure#0}::{closure#0}]>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/coop.rs:98:5
    = note: inside closure at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/runtime/basic_scheduler.rs:184:35
    = note: inside closure at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/runtime/basic_scheduler.rs:266:29
    = note: inside `tokio::macros::scoped_tls::ScopedKey::<tokio::runtime::basic_scheduler::Context>::set::<[closure@tokio::runtime::basic_scheduler::enter<[closure@tokio::runtime::basic_scheduler::Inner<tokio::runtime::driver::Driver>::block_on<std::pin::Pin<&mut std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>>>::{closure#0}], tokio::sync::MutexGuard<std::string::String>, tokio::runtime::driver::Driver>::{closure#0}], tokio::sync::MutexGuard<std::string::String>>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/macros/scoped_tls.rs:61:9
    = note: inside `tokio::runtime::basic_scheduler::enter::<[closure@tokio::runtime::basic_scheduler::Inner<tokio::runtime::driver::Driver>::block_on<std::pin::Pin<&mut std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>>>::{closure#0}], tokio::sync::MutexGuard<std::string::String>, tokio::runtime::driver::Driver>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/runtime/basic_scheduler.rs:266:5
    = note: inside `tokio::runtime::basic_scheduler::Inner::<tokio::runtime::driver::Driver>::block_on::<std::pin::Pin<&mut std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>>>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/runtime/basic_scheduler.rs:176:9
    = note: inside `tokio::runtime::basic_scheduler::InnerGuard::<tokio::runtime::driver::Driver>::block_on::<std::pin::Pin<&mut std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>>>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/runtime/basic_scheduler.rs:405:9
    = note: inside `tokio::runtime::basic_scheduler::BasicScheduler::<tokio::runtime::driver::Driver>::block_on::<std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/runtime/basic_scheduler.rs:136:24
    = note: inside `tokio::runtime::Runtime::block_on::<std::future::from_generator::GenFuture<[static generator@tokio::sync::Mutex<std::string::String>::lock::{closure#0} for<'r, 's> {std::future::ResumeTy, &'r tokio::sync::Mutex<std::string::String>, impl futures::Future, ()}]>>` at /home/alice/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.0.1/src/runtime/mod.rs:450:46
note: inside `main` at src/main.rs:17:5
   --> src/main.rs:17:5
    |
17  |     rt.block_on(m.lock());
    |     ^^^^^^^^^^^^^^^^^^^^^
    = note: inside `<fn() as std::ops::FnOnce<()>>::call_once - shim(fn())` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
    = note: inside `std::sys_common::backtrace::__rust_begin_short_backtrace::<fn(), ()>` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125:18
    = note: inside closure at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:66:18
    = note: inside `std::ops::function::impls::<impl std::ops::FnOnce<()> for &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>::call_once` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/function.rs:259:13
    = note: inside `std::panicking::r#try::do_call::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:379:40
    = note: inside `std::panicking::r#try::<i32, &dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe>` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panicking.rs:343:19
    = note: inside `std::panic::catch_unwind::<&dyn std::ops::Fn() -> i32 + std::marker::Sync + std::panic::RefUnwindSafe, i32>` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/panic.rs:396:14
    = note: inside `std::rt::lang_start_internal` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:51:25
    = note: inside `std::rt::lang_start::<()>` at /home/alice/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/std/src/rt.rs:65:5

error: aborting due to previous error

[hawkw]

yeah, i think you're right that the waiter should be in an UnsafeCell. this should actually be pretty straightforward to fix, i think. good catch, @kprotty!

[blasrodri]

yeah, i think you're right that the waiter should be in an UnsafeCell. this should actually be pretty straightforward to fix, i think. good catch, @kprotty!

Tried following your guide, and still fails. What am I missing?

diff --git a/tokio/src/sync/batch_semaphore.rs b/tokio/src/sync/batch_semaphore.rs
index 803f2a18..49d562bc 100644
--- a/tokio/src/sync/batch_semaphore.rs
+++ b/tokio/src/sync/batch_semaphore.rs
@@ -20,6 +20,7 @@ use crate::loom::sync::atomic::AtomicUsize;
 use crate::loom::sync::{Mutex, MutexGuard};
 use crate::util::linked_list::{self, LinkedList};
 
+use std::cell::UnsafeCell as UnsafeCellSTD;
 use std::future::Future;
 use std::marker::PhantomPinned;
 use std::pin::Pin;
@@ -65,7 +66,7 @@ pub enum TryAcquireError {
 pub struct AcquireError(());
 
 pub(crate) struct Acquire<'a> {
-    node: Waiter,
+    node: UnsafeCellSTD<Waiter>,
     semaphore: &'a Semaphore,
     num_permits: u32,
     queued: bool,
@@ -458,7 +459,7 @@ impl Future for Acquire<'_> {
 impl<'a> Acquire<'a> {
     fn new(semaphore: &'a Semaphore, num_permits: u32) -> Self {
         Self {
-            node: Waiter::new(num_permits),
+            node: UnsafeCellSTD::new(Waiter::new(num_permits)),
             semaphore,
             num_permits,
             queued: false,
@@ -476,7 +477,7 @@ impl<'a> Acquire<'a> {
 
             let this = self.get_unchecked_mut();
             (
-                Pin::new_unchecked(&mut this.node),
+                Pin::new_unchecked(this.node.get_mut()),
                 &this.semaphore,
                 this.num_permits,
                 &mut this.queued,
@@ -499,11 +500,11 @@ impl Drop for Acquire<'_> {
         let mut waiters = self.semaphore.waiters.lock();
 
         // remove the entry from the list
-        let node = NonNull::from(&mut self.node);
+        let node = NonNull::from(self.node.get_mut());
         // Safety: we have locked the wait list.
         unsafe { waiters.queue.remove(node) };
 
-        let acquired_permits = self.num_permits as usize - self.node.state.load(Acquire);
+        let acquired_permits = self.num_permits as usize - self.node.get_mut().state.load(Acquire);
         if acquired_permits > 0 {
             self.semaphore.add_permits_locked(acquired_permits, waiters);
         }

[Darksonn]

Further analysis has showed that it is not so simple.

[kprotty]

@blasrodri The usage of get_mut on the UnsafeCell there looks like an issue. It becomes more vivid where an atomic load, which implies shared references, is being done on a mutable reference.

Worst case (just speculation) this might be more fundamental of a change. With the existence of Pin<&mut Self> from the Future trait, and the need to have other threads performing updates concurrently on the memory of something in that Pin<&mut Self>, you would need to have that thread-shared memory say to the compiler that "a mutable reference and immutable references to this is not UB" which UnsafeCell doesn't seem to currently denote. This would also imply that things like futures-intrusive are unsound as well.

Fwiw, this pattern also appears in sync::Notify iirc so it may not be only this Future type to address. One way to handle this is by heap allocating the shared state in order to keep the "only shared references" property in the face of Pin<&mut Self>. While this addresses it, its a poor compromise to introduce runtime overhead simply due to not enough language support.

[Darksonn]

Relevant thread: Are sound self-referential structs possible without boxing?

The answer appears to be "No, it's not possible".

[Darksonn]

Miri should stop complaining about this once #4397 is merged.

[kprotty]

FWIW, It's possible to fix this soundness issue while still using intrusive Futures. You just need to store the intrusive Node outside the Future that's being polled, then have said Future reference it. This avoids the Pin<&mut Self> transitively creating a mutable reference to the intrusive Node. In reality however, the Node would still be stored in the async fn compiler's generated Future which would still have this issue of transitive pin mut ref when polled but at least its outside user written code and now a fundamental issue with Rust async.

[Darksonn]

In general, we are waiting for a resolution to the problem where async Rust itself is unsound. I don't think what you suggested changes anything regarding what miri will accept.

[udoprog]

Note that JoinSet also trivially triggers miri for me right now:

let mut join_set = JoinSet::new();    
join_set.spawn(async move {});
join_set.join_next().await;

Verbose errors

error: Undefined Behavior: trying to retag from <200248> for SharedReadWrite permission at alloc75572[0x0], but that tag does not exist in the borrow stack for this location
   --> C:\Users\udoprog\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\non_null.rs:376:18
    |
376 |         unsafe { &*self.as_ptr() }
    |                  ^^^^^^^^^^^^^^^
    |                  |
    |                  trying to retag from <200248> for SharedReadWrite permission at alloc75572[0x0], but that tag does not exist in the borrow stack for this location
    |                  this error occurs as part of retag at alloc75572[0x0..0x38]
    |
    = help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
    = help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information   
help: <200248> was created by a SharedReadOnly retag at offsets [0x31..0x81]
   --> tests\test_drop.rs:11:5
    |
11  |     Ok(())
    |     ^^^^^^
    = note: BACKTRACE (of the first span):
    = note: inside `std::ptr::NonNull::<alloc::sync::ArcInner<tokio::util::idle_notified_set::ListEntry<tokio::task::JoinHandle<()>>>>::as_ref::<'_>` at C:\Users\udoprog\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\ptr\non_null.rs:376:18: 376:33
    = note: inside `std::sync::Arc::<tokio::util::idle_notified_set::ListEntry<tokio::task::JoinHandle<()>>>::inner` at C:\Users\udoprog\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\sync.rs:1251:18: 1251:35        
    = note: inside `<std::sync::Arc<tokio::util::idle_notified_set::ListEntry<tokio::task::JoinHandle<()>>> as std::clone::Clone>::clone` at C:\Users\udoprog\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\alloc\src\sync.rs:1501:24: 1501:36
    = note: inside `tokio::util::idle_notified_set::IdleNotifiedSet::<tokio::task::JoinHandle<()>>::pop_notified` at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\util\idle_notified_set.rs:193:30: 193:43
    = note: inside `tokio::task::JoinSet::<()>::poll_join_next` at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\task\join_set.rs:368:31: 368:66
    = note: inside closure at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\task\join_set.rs:284:37: 284:60
    = note: inside `<tokio::future::poll_fn::PollFn<[closure@tokio::task::JoinSet<()>::join_next::{closure#0}::{closure#0}]> as std::future::Future>::poll` at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\future\poll_fn.rs:58:9: 58:19
    = note: inside closure at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\task\join_set.rs:284:62: 284:67
note: inside closure
   --> tests\test_drop.rs:10:26
    |
10  |     join_set.join_next().await;
    |                          ^^^^^
    = note: inside `<std::pin::Pin<&mut dyn std::future::Future<Output = std::result::Result<(), anyhow::Error>>> as std::future::Future>::poll` at C:\Users\udoprog\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\future\future.rs:125:9: 125:61
    = note: inside `<std::pin::Pin<&mut std::pin::Pin<&mut dyn std::future::Future<Output = std::result::Result<(), anyhow::Error>>>> as std::future::Future>::poll` at C:\Users\udoprog\.rustup\toolchains\nightly-x86_64-pc-windows-msvc\lib\rustlib\src\rust\library\core\src\future\future.rs:125:9: 125:61
    = note: inside closure at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\runtime\scheduler\current_thread.rs:541:57: 541:86
    = note: inside `tokio::runtime::coop::with_budget::<std::task::Poll<std::result::Result<(), anyhow::Error>>, [closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut std::pin::Pin<&mut dyn std::future::Future<Output 
= std::result::Result<(), anyhow::Error>>>>>::{closure#0}::{closure#0}::{closure#0}]>` at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\runtime\coop.rs:107:5: 107:8
    = note: inside `tokio::runtime::coop::budget::<std::task::Poll<std::result::Result<(), anyhow::Error>>, [closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut std::pin::Pin<&mut dyn std::future::Future<Output = std::result::Result<(), anyhow::Error>>>>>::{closure#0}::{closure#0}::{closure#0}]>` at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\runtime\coop.rs:73:5: 73:38
    = note: inside closure at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\runtime\scheduler\current_thread.rs:541:25: 541:87
    = note: inside `tokio::runtime::scheduler::current_thread::Context::enter::<std::task::Poll<std::result::Result<(), anyhow::Error>>, [closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut std::pin::Pin<&mut dyn std::future::Future<Output = std::result::Result<(), anyhow::Error>>>>>::{closure#0}::{closure#0}]>` at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\runtime\scheduler\current_thread.rs:350:19: 350:22
    = note: inside closure at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\runtime\scheduler\current_thread.rs:540:36: 542:23
    = note: inside closure at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\runtime\scheduler\current_thread.rs:615:57: 615:79
    = note: inside `tokio::macros::scoped_tls::ScopedKey::<tokio::runtime::scheduler::current_thread::Context>::set::<[closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::enter<[closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut std::pin::Pin<&mut dyn std::future::Future<Output = std::result::Result<(), anyhow::Error>>>>>::{closure#0}], std::option::Option<std::result::Result<(), anyhow::Error>>>::{closure#0}], (std::boxed::Box<tokio::runtime::scheduler::current_thread::Core>, std::option::Option<std::result::Result<(), anyhow::Error>>)>` at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\macros\scoped_tls.rs:61:9: 61:12
    = note: inside `tokio::runtime::scheduler::current_thread::CoreGuard::<'_>::enter::<[closure@tokio::runtime::scheduler::current_thread::CoreGuard<'_>::block_on<std::pin::Pin<&mut std::pin::Pin<&mut dyn std::future::Future<Output = std::result::Result<(), anyhow::Error>>>>>::{closure#0}], std::option::Option<std::result::Result<(), anyhow::Error>>>` at C:\Users\udoprog\.cargo\registry\src\index.crates.io-6f17d22bba15001f\tokio-1.28.1\src\runtime\scheduler\current_thread.rs:615:27: 615:80

[Darksonn]

@udoprog That error is unrelated to what's discussed in this issue. I filed #5693.