[batch] Some tasks are not properly closed on instance shutdown

[daniel-goldstein]

What happened?

Upon worker shutdown, we warn errors if there are any unclosed asyncio Tasks before the process exits. Lately, the worker logs have been noisy with such warnings. See here for an example. We should investigate and make sure all tasks are properly closed so that workers can shutdown gracefully. This may also be relevant to #14261, as the running theory is that the nonsensical assertion error is a byproduct of trying to log after the python process is partially torn down. Ideally, we shouldn't have anything left to log once the process tries to exit but this might happen if there are unjoined tasks.

Version

0.2.128

Relevant log output

No response