Build Runs Every Time Despite Using Remote Cache

[aryeh-looker]

We are doing the following on CI:

    bazel build //... \
      --remote_cache=https://storage.googleapis.com/<path-to-bucket> \
      --google_default_credentials

We have a build target that uses rules docker that bazel runs every time, despite using the remote cache. It does not depend on other targets in our application, except for ones that are rarely modified. Is this expected behavior? It's not completely clear to me whether this is a bug. Any guidance on a workaround or otherwise would be appreciated. Unfortunately, the docker build target is expensive, as it involves compilation of third-party sources. We are looking at sharding the build on CI to multiple machines due to this issue.

[codersasha]

We're having the same issue. Bazel doesn't even seem to detect that it's a cacheable target - it reports a 100% cache hit rate.

[mikasun7]

i found ImageLayer is not "remote_cacheable"，from the execution log(with option --execution_log_binary_file)

remote-cached:

listed_outputs: "bazel-out/k8-fastbuild/bin/apps/fc/ifbook/srcs/pubsub/[email protected]"
remotable: true
cacheable: true
progress_message: "SHA256 apps/fc/ifbook/srcs/pubsub/[email protected]"
mnemonic: "SHA256"
actual_outputs {
  path: "bazel-out/k8-fastbuild/bin/apps/fc/ifbook/srcs/pubsub/[email protected]"
  digest {
    hash: "55f9eac53ee92f92f3afefa32e1a9cecff3a98397df2ce81826b495a1e66209a"
    size_bytes: 64
    hash_function_name: "SHA-256"
  }
}
runner: "remote cache hit"
remote_cache_hit: true
remote_cacheable: true   # <--- here
walltime {
}

not remote-cached(remote_cacheable disappeared):

listed_outputs: "bazel-out/k8-fastbuild/bin/apps/fc/ifbook/srcs/pubsub/[email protected]"
remotable: true
cacheable: true
progress_message: "ImageLayer apps/fc/ifbook/srcs/pubsub/[email protected]"
mnemonic: "ImageLayer"
actual_outputs {
  path: "bazel-out/k8-fastbuild/bin/apps/fc/ifbook/srcs/pubsub/[email protected]"
  digest {
    hash: "664346b8273e5c46f88c70cf3717bf01a01a3f32d65472b6448058962bdd6109"
    size_bytes: 27709440
    hash_function_name: "SHA-256"
  }
}
runner: "processwrapper-sandbox"
walltime {
  nanos: 138000000
}

ImageLayer code: https://github.com/bazelbuild/rules_docker/blob/master/container/layer.bzl#L170