Add CAS indexer for worker re-indexing
Summary:
Over time, buildfarm workers become inaccessible due to termination via scaling or unexpected errors. The redis CAS keys need their references to these workers re-indexed in order to properly reflect the currently active pool of workers. This is primarily an optimization, as stale references to workers can lead to performance issues. We integrate an indexer to run periodically as part of buildfarm's normal operation.
The indexer runs on the server, and a global lock is used within redis.
Note:
This functionality was ported from an internal python script to Java.
@werkt can we have this one merged? We want to test buildfarm in gke with HPA and this seems like a thing we will hit a lot.