[Bazel CI] Error unexpected keyword argument 'incompatible_use_toolchain_transition'

[sgowroji]

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4745#01977c12-fddf-4d54-999a-0ff0d117eebd

Platform: Ubuntu

Logs:

(04:09:29) ERROR: Traceback (most recent call last):
--
  | File "/var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/3025b0a37590b9c7d740f1ac50d854c9/external/aspect_bazel_lib++toolchains+bsd_tar_toolchains/defs.bzl", line 15, column 20, in <toplevel>
  | resolved_toolchain = rule(
  | Error in rule: rule() got unexpected keyword argument 'incompatible_use_toolchain_transition'
  | (04:09:29) ERROR: /var/lib/buildkite-agent/builds/bk-docker-j6js/bazel-org-repo-root/bazel-remote/BUILD.bazel:104:10: While resolving toolchains for target //:bazel-remote-image (3a46b04): invalid registered toolchain '@bsd_tar_toolchains//:all': while parsing '@bsd_tar_toolchains//:all': error loading package '@@aspect_bazel_lib++toolchains+bsd_tar_toolchains//': initialization of module 'defs.bzl' failed
  | (04:09:29) ERROR: Analysis of target '//:bazel-remote-image' failed; build aborted
  | (04:09:29) INFO: Elapsed time: 2.776s, Critical Path: 0.02s
  | (04:09:29) INFO: 1 process: 1 internal.
  | (04:09:29) ERROR: Build did NOT complete successfully
  | (04:09:29) FAILED:

Culprit: https://github.com/bazelbuild/bazel/commit/f53608ac3dbfd02cecb4bb78a138badb0d84e311

To Fix this issue need to Remove no-op incompatible_use_toolchain_transition param

Steps:

git clone https://github.com/buchgr/bazel-remote
git reset 845eaf90b5d33ff85bc466a5f7efdb8c44df749e  --hard
export USE_BAZEL_VERSION=35fb97fe269e42be22fcfa81ff6cdfc4b5bc291e 
bazel build //:bazel-remote //:bazel-remote-image

CC Greenteam @fweikert

[mostynb]

@sgowroji: I think the problem is in rules_pkg. Filed https://github.com/bazelbuild/rules_pkg/issues/962 and submitted https://github.com/bazelbuild/rules_pkg/pull/963 to try to fix the problem there.

[mostynb]

The potential fix landed in rules_pkg, waiting for a new release...

[mostynb]

@kotlaja provided a workaround, I think this might be working again now.