rules_python Reimplement rules in Starlark in rules

This is the general tracking issue for re-implementing the Bazel-bundled Python rules in Starlark in the rules_python repo.

The Bazel-side issue is https://github.com/bazelbuild/bazel/issues/15897

As of now (Feb 2023), the Bazel bundled rules (not providers or flags, though) have all been re-implemented in Starlark. They will used in Bazel 7. Hooks for low-level Bazel behaviors the Python rules need are exposed to the rules_python repo. So now this is mostly a matter of copying that code over here, cleaning it up, and flushing out any missing hooks.

TODO

[x] Copy Bazel code over to rules_python
[ ] Re-implement Bazel tests in rules_python
[ ] Re-implement flags in Starlark
[x] Setup a CI config that uses e.g. bazel=rolling to get access to pre-release bazel versions
[x] Enable pystar by default

That's the high level gist, some lower level TODOs are:

[x] Implement compatibility shim so rules_python and Bazel rules can interoperate to some degree
[ ] Deprecate and remove Bazel-bundled flags
[ ] Deprecate and remove Python-related parts in @bazel_tools

An optional bonus thing is to backport the low-level hooks to 5.x and/or 6.x. This makes them available sooner, so that we don't have to wait for Bazel 7. There isn't a ton to backport (I intentionally made them to be amenable to this idea), but there's several small pieces. The two general pieces are exposing the flags as fragments and the py_internal object:

py_internal object and it's backing pieces:
Configuration fields (Bazel builtin flags); Strictly speaking, only flags that don't have a fixed set of values (e.g arbitrary string or labels) need to be made available. Other flags can be "wrapped" in a regular Starlark-defined flag by just having an exhaustive listing of config_settings
- --python_path and --python_top from BazelPythonConfiguration; note, however, that --python_top is supposed to be unused (superceded by toolchains) and --python_path is probably unused (unclear; some comments indicate it might be used as part of windows support)
- --build_python_zip from PythonConfiguration.java; note, however, it, too, can be replicated in Starlark by using platforms or the py_internal.get_current_os_name() helper.

Feb 14 '23 07:02 rickeylev

@rickeylev I can consistently reproduce build failure when running:

USE_BAZEL_VERSION=last_green RULES_PYTHON_ENABLE_PYSTAR=1 bazelisk test //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test
2023/10/17 09:12:13 Using unreleased version at commit 14e3d1b825dab76c5886fffb9dd1c01841c18795
2023/10/17 09:12:13 Downloading https://storage.googleapis.com/bazel-builds/artifacts/centos7/14e3d1b825dab76c5886fffb9dd1c01841c18795/bazel...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Analyzed target //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test (69 packages loaded, 3152 targets configured).
FAIL: //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test (see /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/execroot/rules_python/bazel-out/k8-fastbuild/testlogs/python/tests/toolchains/python_3_9_10_x86_64-unknown-linux-gnu_test/test.log)
INFO: From Testing //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test:
==================== Test output for //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test:
$TEST_TMPDIR defined: output root default is '/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4' and max_idle_secs default is '15'.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/bazel-out/k8-fastbuild/bin/python/tests/toolchains/run_test_3.9.10.sh.runfiles/rules_python/python/tests/toolchains/3.9.10/.bazelrc:
  Inherited 'build' options: --override_repository rules_python=/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/bazel-out/k8-fastbuild/bin/python/tests/toolchains/run_test_3.9.10.sh.runfiles/rules_python --test_output=errors
Computing main repo mapping:
WARNING: --enable_bzlmod is set, but no MODULE.bazel file was found at the workspace root. Bazel will create an empty MODULE.bazel file. Please consider migrating your external dependencies from WORKSPACE to MODULE.bazel. For more details, please refer to https://github.com/bazelbuild/bazel/issues/18958.
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
Loading:
Loading: 0 packages loaded
Analyzing: target @python//:python3 (1 packages loaded, 0 targets configured)
Analyzing: target @python//:python3 (1 packages loaded, 0 targets configured)
[0 / 1] checking cached actions
Analyzing: target @python//:python3 (2 packages loaded, 0 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
INFO: Analyzed target @python//:python3 (5 packages loaded, 16 targets configured).
INFO: Found 1 target...
Target @python_x86_64-unknown-linux-gnu//:bin/python3 up-to-date (nothing to build)
INFO: Elapsed time: 22.738s, Critical Path: 0.07s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4/_bazel_aignas/485f80f93d904973eb69e2d94ad629f9/external/python_x86_64-unknown-linux-gnu/bin/python3 --version
$TEST_TMPDIR defined: output root default is '/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4' and max_idle_secs default is '15'.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'test' from /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/bazel-out/k8-fastbuild/bin/python/tests/toolchains/run_test_3.9.10.sh.runfiles/rules_python/python/tests/toolchains/3.9.10/.bazelrc:
  Inherited 'build' options: --override_repository rules_python=/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/bazel-out/k8-fastbuild/bin/python/tests/toolchains/run_test_3.9.10.sh.runfiles/rules_python --test_output=errors
Computing main repo mapping:
Loading:
Loading: 0 packages loaded
Analyzing: target //:python_version_test (1 packages loaded, 0 targets configured)
Analyzing: target //:python_version_test (1 packages loaded, 0 targets configured)
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Analyzing: target //:python_version_test (50 packages loaded, 2 targets configured)
[1 / 1] checking cached actions
Analyzing: target //:python_version_test (57 packages loaded, 169 targets configured)
[1 / 1] checking cached actions
Analyzing: target //:python_version_test (57 packages loaded, 169 targets configured)
[1 / 1] checking cached actions
Analyzing: target //:python_version_test (64 packages loaded, 507 targets configured)
[1 / 1] checking cached actions
Analyzing: target //:python_version_test (64 packages loaded, 507 targets configured)
[1 / 1] checking cached actions
INFO: Analyzed target //:python_version_test (70 packages loaded, 544 targets configured).
[6 / 7] Testing //:python_version_test; 0s linux-sandbox
FAIL: //:python_version_test (see /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4/_bazel_aignas/485f80f93d904973eb69e2d94ad629f9/execroot/_main/bazel-out/k8-fastbuild/testlogs/python_version_test/test.log)
INFO: From Testing //:python_version_test:
==================== Test output for //:python_version_test:
F
======================================================================
FAIL: test_match_toolchain (__main__.TestPythonVersion.test_match_toolchain)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4/_bazel_aignas/485f80f93d904973eb69e2d94ad629f9/sandbox/linux-sandbox/1/execroot/_main/bazel-out/k8-fastbuild/bin/python_version_test.runfiles/_main/python_version_test.py", line 22, in test_match_toolchain
    self.assertEqual(platform.python_version(), os.getenv("PYTHON_VERSION"))
AssertionError: '3.11.5' != '3.9.10'
- 3.11.5
+ 3.9.10


----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)
================================================================================
INFO: Found 1 test target...
Target //:python_version_test up-to-date:
  bazel-bin/python_version_test
INFO: Elapsed time: 7.701s, Critical Path: 0.60s
INFO: 7 processes: 5 internal, 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 7 total actions
//:python_version_test                                                   FAILED in 0.4s
  /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4/_bazel_aignas/485f80f93d904973eb69e2d94ad629f9/execroot/_main/bazel-out/k8-fastbuild/testlogs/python_version_test/test.log

Executed 1 out of 1 test: 1 fails locally.
E
======================================================================
ERROR: test_match_toolchain (__main__.TestPythonVersion.test_match_toolchain)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/bazel-out/k8-fastbuild/bin/python/tests/toolchains/run_test_3.9.10.sh.runfiles/rules_python/python/tests/toolchains/3.9.10/run_acceptance_test.py", line 71, in test_match_toolchain
    subprocess.run("bazel test --announce_rc //...", shell=True, check=True)
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/execroot/rules_python/external/python_3_11_6_x86_64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'bazel test --announce_rc //...' returned non-zero exit status 3.

----------------------------------------------------------------------
Ran 1 test in 30.865s

FAILED (errors=1)
================================================================================

It seems to resolve the wrong version of the toolchain and I think this is something to have changed in the bazel@HEAD.

Oct 17 '23 00:10 aignas

Doing a bisect:

$ RULES_PYTHON_ENABLE_PYSTAR=1 bazelisk --bisect=2c9d68e05861ac3e71cb86de2b69bc52403411bf..HEAD test //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test

yielded the following known to work commits:

https://github.com/bazelbuild/bazel/commit/75657569220178f11ec12f7ce33812e571c6c854 (passes)
https://github.com/bazelbuild/bazel/commit/2b8219042c (passes)

Then there is a series of commits that are failing:

https://github.com/bazelbuild/bazel/commit/659d7b223a (broken build)
https://github.com/bazelbuild/bazel/commit/30d033cd7e (broken build)
https://github.com/bazelbuild/bazel/commit/12bc1e6c27 (broken build)
https://github.com/bazelbuild/bazel/commit/0e1b3e2d40 (broken build)
https://github.com/bazelbuild/bazel/commit/bc1df91807 (broken build)
https://github.com/bazelbuild/bazel/commit/cadbaa5626 (broken build)
https://github.com/bazelbuild/bazel/commit/137d3f18be (broken build)
https://github.com/bazelbuild/bazel/commit/4eada724a3 (test is failing)

Oct 17 '23 00:10 aignas

Create bazelbuild/bazel#1496 for the current CI issues.

Oct 17 '23 04:10 aignas

wondering if pyc support is on the roadmap? https://github.com/bazelbuild/rules_python/issues/1761

Dec 29 '23 18:12 Ryang20718

The 0.31.0 release of rules_python has enabled the rules_python Starlark implementation for Bazel 7+. :fireworks: !

If you encounter issues, please file bugs in rules_python. I'm eager to identify and fix any issues. If you do encounter issues, setting RULES_PYTHON_ENABLE_PYSTAR=0 in your environment (directly or using --action_env) will disable the rules_python implementation.

The most likely problem to occur is using a combination of the Bazel-builtin rules/providers and the rules_python rules/providers. They should interoperate OK, but please report any issues.

Feb 13 '24 17:02 rickeylev

Let's close this as complete as it is working well. Thank you @rickeylev!

Jun 17 '24 23:06 aignas

Reimplement rules in Starlark in rules_python