Docs refer to stale bazel-provided images, new images aren't compatible with Cloud Build

[hulkholden]

A couple of the READMEs refer to gcr.io/cloud-marketplace-containers/google/bazel as alternative images provided by the bazel team, but it looks like this repo hasn't been updated since 2020.

The bazel docs now point at gcr.io/bazel-public/bazel which does seem to be up-to-date.

However, I was unable to get these images to work as a drop-in replacement with Cloud Build.

Small example:

steps:
- id: 'Bazel version'
  name: 'gcr.io/bazel-public/bazel:6.2.1'
  entrypoint: 'bazel'
  args: ['version']

Gives:

Step #0 - "Bazel version": Status: Downloaded newer image for [gcr.io/bazel-public/bazel:6.2.1](http://gcr.io/bazel-public/bazel:6.2.1)
Step #0 - "Bazel version": [gcr.io/bazel-public/bazel:6.2.1](http://gcr.io/bazel-public/bazel:6.2.1)
Step #0 - "Bazel version": FATAL: mkdir('/builder/home/.cache/bazel/_bazel_ubuntu'): (error: 13): Permission denied
Finished Step #0 - "Bazel version"

I think this is because the bazel Dockerfile is setting up an ubuntu user rather than running as root. Another consequence of this is that /workspace/ is not writeable which makes it painful to pass around outputs between build steps.

One workaround for the cache issue is to pass --output_user_root like this:

steps:
- id: 'Bazel version'
  name: 'gcr.io/bazel-public/bazel:6.2.1'
  entrypoint: 'bazel'
  args: ['--output_user_root=/home/ubuntu/.bazel', 'version']

But that still doesn't resolve the /workspace/ issues.

So how should the cloud-builders READMEs be updated? I can send a PR to update the stale reference to gcr.io/bazel-public/bazel, but as the images are not drop-in replacements I think it would cause more confusion.

Maybe https://github.com/GoogleCloudPlatform/cloud-builders/blob/master/bazel/README.md can provide a bit more detail and the top-level README could point there?

Maybe there's a deeper compatibility issue which needs resolving? It is surprising that two Google-maintained bits of build infrastructure don't play nicely together.

[t-hale]

Oh man - I've been banging my head on this for ~a week~ much longer than I'm willing to admit. I played around with --output_user_root, but couldn't get to a good solution that I liked. Did you ever ended up figuring this one out? How did you address it? I've tried both 6.x and 7.x versions - same issue for both

[hulkholden]

Hey @t-hale - I forgot most of the details already, but IIRC https://github.com/GoogleCloudPlatform/cloud-builders/pull/925 provided a gcr.io/cloud-builders/bazel:6.2.1 image which worked as expected.

You can take a look at https://github.com/hulkholden/words/blob/main/cloudbuild.yaml to see the cloudbuild workflow that worked for me.