Add nowait option to --remote_cache_async

[brianduff]

This PR updates --remote_cache_async flag to support a new nowait mode that controls when Bazel waits for remote cache uploads to complete (similar to --bes_upload_mode).

Motivation

Currently, Bazel blocks at the end of every build waiting for all remote cache uploads to complete. For builds with large outputs, or on slow connections, this can add significant time to the build even though the uploads don't affect build correctness. Users who want faster build completion times should be able to defer this waiting.

Implementation

• The build completes as soon as action execution finishes
• Uploads continue in the background
• The next command waits up to 30 seconds for uploads to complete
• Server shutdown (bazel shutdown) also waits up to 30 seconds
• Temp files and RPC logs are cleaned up via a future listener after uploads complete

Testing

• Added integration tests in RemoteUploadModeIntegrationTest.java
• Added unit tests for shutdown(RemoteUploadMode) in RemoteExecutionServiceTest.java
• Manually verified async uploads work correctly
• Verified interruption behavior via Ctrl-C.

Closes #14638

[bharadwaj08-one]

@brianduff Could you please take a look at the failing checks?

[brianduff]

Thanks for the feedback so far. I'll take a look at the failing checks next.

[brianduff]

I think this is ready to be reviewed. Thanks!

[brianduff]

Thanks for addressing the comments, this looks pretty good!

How does the added asynchronicity interact with the BES? I recently sent a change that delayed the closure of the stream until the async uploads were done, but I'm not entirely sure what that logic would do in the fully async case.

cc @tjgq @coeuvre

I've experimented with all combinations of the various modes for BES and remote upload as well as interrupting them in both wait and no wait mode, and haven't seen any unusual behavior. Is there a specific thing I should test / look out for?

[brianduff]

OK, I was able to get a message to print! I had to add a new lifecycle method to BlazeModule to facilitate this, but I think the experience is far better.

[brianduff]

Looks good to me, thanks! It's likely that Bazel team members will only be able to review this for approval in the new year though.

Understood. Thanks for the review! :)