add tail tasks API and use for remote cache writes

[tdyas]

Add the concept of "tail tasks" to Pants sessions so that Rust code can schedule async tasks for execution where the tasks should complete before a particular Pants run is considered complete.

First use case: Schedule remote cache writes as tail tasks.

[tdyas]

cc @somdoron

[tdyas]

Just discovered tokio::task::JoinSet which seems relevant.

[tdyas]

There has been some significant refactoring of this PR. It is worth a re-review.

[tdyas]

The latest commits introduce a TailTasks time which encapsulate both spawning and waiting for tail tasks.

[tdyas]

It appears that log_cache_error is not working with tail tasks. https://github.com/pantsbuild/pants/actions/runs/3144144927/jobs/5110386492#step:13:598

I can add regular logging to show that the tail tasks are completing, but errors from log_cache_error are not being displayed.

[stuhood]

It appears that log_cache_error is not working with tail tasks. https://github.com/pantsbuild/pants/actions/runs/3144144927/jobs/5110386492#step:13:598

I can add regular logging to show that the tail tasks are completing, but errors from log_cache_error are not being displayed.

Mmm... that would be because the task local / thread local information is not being propagated to the new task. See the use of Self::future_with_correct_context here: https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/rust/engine/task_executor/src/lib.rs#L121-L137

[tdyas]

Mmm... that would be because the task local / thread local information is not being propagated to the new task. See the use of Self::future_with_correct_context here:

https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/rust/engine/task_executor/src/lib.rs#L121-L137

That was it. Thanks!

It apparently worked in an earlier iteration of the PR only due to the double-spawn, one of which was via Executor which did wrap the task with future_with_correct_context.