bazel-buildfarm icon indicating copy to clipboard operation
bazel-buildfarm copied to clipboard

Add CAS indexer for worker re-indexing

Open luxe opened this issue 5 years ago • 1 comments

Summary:

Over time, buildfarm workers become inaccessible due to termination via scaling or unexpected errors. The redis CAS keys need their references to these workers re-indexed in order to properly reflect the currently active pool of workers. This is primarily an optimization, as stale references to workers can lead to performance issues. We integrate an indexer to run periodically as part of buildfarm's normal operation.

The indexer runs on the server, and a global lock is used within redis.

Note:

This functionality was ported from an internal python script to Java.

luxe avatar Jun 17 '20 15:06 luxe

@werkt can we have this one merged? We want to test buildfarm in gke with HPA and this seems like a thing we will hit a lot.

cristifalcas avatar Jul 10 '20 13:07 cristifalcas