Please make Jedis / Redis replaceable

[walles]

trafficstars

It would be nice if the backplane was pluggable.

Best case the instructions for adding a new backplane should be:

1. Implement one interface
2. Register your interface somehow
3. Done!

Here's one experiment indicating that the backplane is not pluggable:

~/s/b/src-bazel-buildfarm (master|✔) $ rg jedis -l|grep -v /redis/
src/main/protobuf/build/buildfarm/v1test/buildfarm.proto
src/test/java/build/buildfarm/BUILD
src/test/java/build/buildfarm/instance/shard/RedisShardBackplaneTest.java
src/test/java/build/buildfarm/instance/shard/RedisShardSubscriberTest.java
defs.bzl
deps.bzl
examples/shard-server.config.example
examples/shard-worker.config.example
third_party/jedis/BUILD
src/main/java/build/buildfarm/BUILD
src/main/java/build/buildfarm/WorkerProfile.java
src/main/java/build/buildfarm/instance/shard/JedisClusterFactory.java
src/main/java/build/buildfarm/instance/shard/RedisShardSubscriber.java
src/main/java/build/buildfarm/instance/shard/RedisShardSubscription.java
src/main/java/build/buildfarm/instance/shard/RedisShardBackplane.java
src/main/java/build/buildfarm/instance/shard/OperationQueue.java
src/main/java/build/buildfarm/common/WorkerIndexer.java
src/main/java/build/buildfarm/operations/EnrichedOperationBuilder.java
src/main/java/build/buildfarm/operations/OperationsFinder.java
~/s/b/src-bazel-buildfarm (master|✔) $

[luxe]

Throwing in my 2 cents: The majority of the jedis usage is behind RedisShardBackplane. In theory, implementing that interface would get you 90% of the way there. However, the backplane itself is large and requires a lot of buildfarm specific logic.

The most feasible path I see to unlocking a new backplane is by first simplifying the existing one. If all jedis usage becomes isolated into commonly understood data structures, it would become trivial to replace those data structures with ones that use a different backend.

This is the refactoring path I've been taking, and the backplane is partially like this now.

Many of the files you posted are either used by the backplane already (like the operationQueue) or optional features you can ignore when using another backend. WorkerIndexer / WorkerProfile would be disabled. The files under operation/ are for finding operations which is an API only called by users.

But anyways, it's not as plug-and-play as you say. There is work to be done here! :)

[walles]

If all jedis usage becomes isolated into commonly understood data structures, it would become trivial to replace those data structures with ones that use a different backend.

This is the refactoring path I've been taking, and the backplane is partially like this now.

👍 👍

[werkt]

There's a hidden partition waiting to happen that could substantively divide this task and the interface:

ShardBackplane (which should be renamed to Backplane) is really three separate interfaces, and they don't cross at all: ContentAddressableStorageIndex (not yet defined) ActionCache (already defined) Execution (not yet defined)

The last one is more expansive and poorly distinguished than the rest, but I think as a first stab, doing the separation of CAS and AC, because they would represent a usable --remote_cache target, would be valuable. I'm happy to leave a 'full-featured' interface that is an aggregate of all three for existing Shard usage.