pants usually miss the last remote cache write

[somdoron]

Describe the bug Pants (without pantsd) usually miss the last remote cache write, probably because the cache write is asynchronous and doesn't prevent the pants process from completing.

Pants version pants_version = "2.14.0rc1"

OS MacOS, Linux

Additional info counters:

  backtrack_attempts: 0
  local_cache_read_errors: 0
  local_cache_requests: 0
  local_cache_requests_cached: 0
  local_cache_requests_uncached: 0
  local_cache_total_time_saved_ms: 0
  local_cache_write_errors: 0
  local_execution_requests: 74
  local_process_total_time_run_ms: 51177
  remote_cache_read_errors: 0
  remote_cache_requests: 1099
  remote_cache_requests_cached: 1082
  remote_cache_requests_uncached: 6
  remote_cache_speculation_local_completed_first: 11
  remote_cache_speculation_remote_completed_first: 1082
  remote_cache_total_time_saved_ms: 312993
  remote_cache_write_attempts: 17
  remote_cache_write_errors: 0
  remote_cache_write_successes: 16 # one is missing here
  remote_execution_errors: 0
  remote_execution_requests: 0
  remote_execution_rpc_errors: 0
  remote_execution_rpc_execute: 0
  remote_execution_rpc_retries: 0
  remote_execution_rpc_wait_execution: 0
  remote_execution_success: 0
  remote_execution_timeouts: 0
  remote_process_total_time_run_ms: 0
  remote_store_missing_digest: 0

[Eric-Arellano]

I believe this would be solved by https://github.com/pantsbuild/pants/issues/11688, which @tdyas has been looking at

[somdoron]

Yes, should I close it as a duplicate?

[Eric-Arellano]

Eh, we can keep those so that we're confident the tail tasks are being used to fix this use case. Thanks!

[tdyas]

I believe this would be solved by #11688, which @tdyas has been looking at

I looked at tail tasks this week primarily in order to solve how cleanly shutdown the Docker command runner. The solution for that use case doesn't really overlap with how "tail tasks" would operate to solve the remote cache write issue.

That said, I do have a stale local branch where I did make some progress many months ago on implementing tail tasks. I'll try to update that branch this week.

[tdyas]

@somdoron: Can you try out https://github.com/pantsbuild/pants/pull/16952?

[somdoron]

Looks good, I run it multiple times: ./pants_from_sources --no-local-cache --no-pantsd --stats-log --no-remote-cache-read --remote-cache-write. Stats below:

remote_cache_write_attempts: 1131
remote_cache_write_errors: 0
remote_cache_write_successes: 1131

[stuhood]

#16952 landed: thanks @tdyas!