pants
pants copied to clipboard
add tail tasks API and use for remote cache writes
Add the concept of "tail tasks" to Pants sessions so that Rust code can schedule async tasks for execution where the tasks should complete before a particular Pants run is considered complete.
First use case: Schedule remote cache writes as tail tasks.
cc @somdoron
Just discovered tokio::task::JoinSet
which seems relevant.
There has been some significant refactoring of this PR. It is worth a re-review.
The latest commits introduce a TailTasks
time which encapsulate both spawning and waiting for tail tasks.
It appears that log_cache_error is not working with tail tasks. https://github.com/pantsbuild/pants/actions/runs/3144144927/jobs/5110386492#step:13:598
I can add regular logging to show that the tail tasks are completing, but errors from log_cache_error
are not being displayed.
It appears that log_cache_error is not working with tail tasks. https://github.com/pantsbuild/pants/actions/runs/3144144927/jobs/5110386492#step:13:598
I can add regular logging to show that the tail tasks are completing, but errors from
log_cache_error
are not being displayed.
Mmm... that would be because the task local / thread local information is not being propagated to the new task. See the use of Self::future_with_correct_context
here: https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/rust/engine/task_executor/src/lib.rs#L121-L137
Mmm... that would be because the task local / thread local information is not being propagated to the new task. See the use of
Self::future_with_correct_context
here:https://github.com/pantsbuild/pants/blob/cdf21fc69954aec999ed57ed988bf67a31148bf9/src/rust/engine/task_executor/src/lib.rs#L121-L137
That was it. Thanks!
It apparently worked in an earlier iteration of the PR only due to the double-spawn, one of which was via Executor
which did wrap the task with future_with_correct_context
.