Remote Downloader: Does not honor --remote_local_fallback

[jgao54]

Description of the bug:

We have --experimental_remote_downloader flag enabled with bazel-remote cache, however it does not honor --remote_local_fallback. When the remote cache shuts down unexpected, the following error is observed:

INFO: Repository io_bazel_rules_go instantiated at:
  /home/circleci/figma/WORKSPACE.bazel:278:13: in <toplevel>
Repository rule http_archive defined at:
  /home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/bazel_tools/tools/build_defs/repo/http.bzl:353:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'io_bazel_rules_go':
   Traceback (most recent call last):
	File "/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/bazel_tools/tools/build_defs/repo/http.bzl", line 100, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
ERROR: /home/circleci/figma/WORKSPACE.bazel:278:13: fetching http_archive rule //external:io_bazel_rules_go: Traceback (most recent call last):
	File "/home/circleci/.cache/bazel/_bazel_circleci/a6e8cb0ce75ceb1ea1faa0cca0b0ae0d/external/bazel_tools/tools/build_defs/repo/http.bzl", line 100, column 45, in _http_archive_impl
		download_info = ctx.download_and_extract(
Error in download_and_extract: io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
ERROR: no such package '@io_bazel_rules_go//go': io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection reset by peer
INFO: Elapsed time: 13.460s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
ERROR: Couldn't start the build. Unable to run tests
FAILED: Build did NOT complete successfully (0 packages loaded)

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1. start bazel-remote with --experimental_remote_asset_api enabled
2. build a target with --experimental_remote_downloader and --remote_local_fallback enabled (can use a build target that uses rules_go, but any repo should work)
3. as soon as you begin to observe remote downloads in bazel-remote logs, send a SIGTERM to the bazel-remote process
4. build will fail with above error

Which operating system are you running Bazel on?

Linux / Macos

What is the output of bazel info release?

release 5.2.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

N/A

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

N/A

Have you found anything relevant by searching the web?

Similar issue as https://github.com/bazelbuild/bazel/issues/12417#issuecomment-741639582 Relevant PR discussion: https://github.com/bazelbuild/bazel/pull/12528

Any other information, logs, or outputs that you want to share?

Agree with this comment, would love to see the option of fall back to local download as a configurable option, regardless of what the default is. Using a separate flag for this (like --experimental_remote_downloader_http_fallback) is reasonable too if the concern is around overloading the existing --remote_local_fallback feature.

[coeuvre]

Now I look into this problem again, I agree with you and me in the past that we should have this configurable. I prefer to have the flag --experimental_remote_downloader_http_fallback to make it more explicit.

I will open #12528 and try to get it merged.