Can not http proxy to another bazel-remote instance

[andrewhamon]

Steps to reproduce

1. Set up two bazel-remote instances, with one http proxying to the other:

bazel remote 1:

bazel-remote --dir /tmp/bazel_remote_1

bazel remote 2:

bazel-remote --dir /tmp/bazel_remote_2 --http_address localhost:8081 --grpc_address localhost:9093 --http_proxy.url http://localhost:8080

2. Run bazel against the second bazel remote:

build --remote_cache=grpc://localhost:9093 //path_to:target

3. Observe tons of failed requests on the first bazel-temote:

It seems like when attempting to perform HTTP proxying, bazel-remote tries to read and write to /cas.v2/, however bazel-remote rejects these as bad requests. All these hashes do already exist on the filesystem, and if I manually do a curl for the same hash but at /cas/, then things work as expected.

Version

Latest release v2.3.3

bazel-remote built with go1.17.5 from git commit 0bc183e484eeb11f403ee1ecea5cf7dd759144bb

Why am I trying this anyway

I'm somewhat interested in setting up a local caching layer for devs at my company, proxying to a shared cache for all developers. This is because a locally running bazel cache, once its cache is filled, is substantially faster.

[mostynb]

Hi, this is a (hopefully temporary) limitation of the default compressed storage mode on the instance 2. If you add --storage_mode uncompressed to instance 2 (and optionally to instance 1) then I think this should work.

At some point I would like to make this work with compressed storage (and transfer compressed blobs between the two instances).

[andrewhamon]

Thanks for the quick reply @mostynb, that makes sense.

This does lead to a few more questions on my part.

The readme says:

Uploaded CAS blobs are stored in a zstandard compressed format by default, which can increase the effective cache size and reduce load on the server if clients also download blobs in zstandard compressed form.

But I noticed that --experimental_remote_cache_compression in bazel defaults to false.

Does that mean that the default --storage_mode zstd actually increases, rather than decreases, server load for bazel clients with default configurations, since the server must decompress from storage (rather than pass the bytes straight through)?

[mostynb]

Does that mean that the default --storage_mode zstd actually increases, rather than decreases, server load for bazel clients with default configurations, since the server must decompress from storage (rather than pass the bytes straight through)?

Yes, that's right- but you can store more data in the cache, which might increase your cache hit rate.

If the clients mostly download compressed data and bazel-remote is using compressed storage, it can just write those bytes directly. If clients upload compressed data bazel-remote still has to decompress it to verify the hash (and it then recompresses to allow downloads from non-zero offsets without decompressing the entire blob).

BTW I don't recommend using bazel with --experimental_remote_cache_compression yet, there's at least one bug that needs fixing first (bazelbuild/remote-apis#212).