bug(quinn): making a connection that never gets to a full `Connection` causes excessive delays in `Endpoint::wait_idle`

[ramfox]

I would appreciate any help.

The bug is illustrated in the endpoint_wait_idle test in draft PR https://github.com/quinn-rs/quinn/pull/2146, but I will also copy the permalink:

https://github.com/quinn-rs/quinn/blob/dd14b01613d7825f9c4ca25c9cd10b076a077aa7/quinn/src/tests.rs#L154-L177

To summarize, if you have an Endpoint that attempts a connection, and for whatever reason, that connection does not make it past the Connecting stage, it causes the Endpoint::wait_idle call (when attempting to gracefully close the endpoint) to always take ~3s. I'm assuming something happens like the ConnectionHandler that is created when Connecting never gets cleaned up and so wait_idle only returns after a timeout.

I'm only familiar enough with quinn to know something is wrong, but not what to do to fix it.