incompatible_default_to_explicit_init_py

[brandjon]

Flag: --incompatible_default_to_explicit_init_py Available since: 1.2 Will be flipped in: ??? Feature tracking issue: #7386

Motivation

For py_binary and py_test targets, Bazel currently automatically creates empty __init__.py files in the runfiles tree wherever it is needed to import Python files. This is done in the ancestor directories of paths that contain Python files, up to but not including the top-level workspace directory, where an explicit __init__.py file must be used. Thus for example, if your py_binary target depends (directly or indirectly) on //pkg/subpkg:foo.py, but your workspace has no //pkg:__init__.py or //pkg/subpkg:__init__.py (or these files were not declared as dependencies), your target will behave as if they exist and are empty.

We want to deprecate this because it is magic at a distance. Python programmers are already used to creating __init__.py files in their source trees, so doing it behind their backs introduces confusion and changes the semantics of imports (since these directories will no longer be considered namespace packages). Eliminating this behavior also should allow us to remove some special runfiles logic.

Change

py_binary and py_test already have a legacy_create_init attribute which effectively defaults to true. With this flag enabled, the effective default becomes false. You can still opt back into true for targets that need it, but in the future we will do another incompatible change to remove the attribute altogether.

I say "effectively" true or false because before this flag was added, the attribute was an actual boolean, and now it is a tristate that defaults to auto, where auto means consult this flag. It is possible some .bzl macro that tries to introspect a py_binary or py_test's attributes dictionary (via native.existing_rules) will observe this change in attribute type, even without enabling the incompatible flag.

Migration

If your build depended on having empty __init__.py files automatically created, you should explicitly create these files in your source tree and add them to the srcs of the relevant py_library targets. If for whatever reason that's not feasible at the moment, you can temporarily opt out of this change on a per-target basis even when the flag itself is enabled, by explicitly adding legacy_create_init = True to your targets.

Timing

It is currently unclear how burdensome migration will be, so we do not yet know when we will flip this flag.

[brandjon]

Blocked by #10098 (among other things).

[HungryOrc]

Hi Brandon, thanks a lot for your time! I've encountered an error of AttributeError: 'module' object has no attribute 'get_world_size' today, when I tried to run xgboost in my python script. I saw Greg's comment here: https://github.com/bazelbuild/rules_python/issues/122, where he mentioned --incompatible_default_to_explicit_init_py, and then I found this Github commit here. I updated my bazel version but still the error exists. Could you offer me a bit more information on how to solve this AttributeError? Thanks a lot!

[brandjon]

I'm not familiar with xgboost, but the comment you linked indicates that enabling this flag solves that issue. Note that this flag is not currently enabled by default (and won't be until some unknown future major Bazel version bump), so you need to pass --incompatible_default_to_explicit_init_py on your bazel build command line (or equivalent bazelrc file).

[HungryOrc]

Thanks a lot Jon!

[limdor]

Blocked by #10098 (among other things).

@brandjon I think it is also block by https://github.com/bazelbuild/bazel/issues/12238

[brandjon]

Unfortunately I never got to implement this when I was maintaining Python rules. Unassigning to get off my issue queue.

[sgowroji]

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.

[fmeum]

@sgowroji This is a tracking issue for an incompatible flag, not stale.

[brentleyjones]

@bazelbuild/triage can we get this triaged? It's still relevant.

[Wyverald]

@rickeylev could you have a look and gauge how much work it would be to flip this?

[justinhorvitz]

Ran into this because my current work is encountering a bit of additional complexity due to the magical appearance of additional empty files when creating runfiles symlink trees.

[rickeylev]

Flipping it within Google is a decent sized project. For that, I'm not the best contact any more; the new team is Munich is responsible now.

Flipping it externally: For the Bazel builtin rules, I'm not sure; those are also the purview of the Munich team. For rules_python, we could do it most anytime (see https://rules-python.readthedocs.io/en/latest/contributing.html#breaking-changes for our breaking changes process)

[zaka634]

can we have some compromised flag instead of this yes or no init file flag ? Propose : If the init.py is present in the python package, just bring it over into the runfile tree, if not present, don't auto-create it. The current default behavior already visit the workspace init.py, it should be pretty easy change to add the present init.py into the runfile tree instead of automatically creating it, thus it create the flexibility to keep the workspace as it is, allow user to control a given package as standard or namespace package, and without adding the burden for user to manually init.py into src/deps everywhere with the potential of circular dependency.

Circular dependency illustration : If we turn off the init auto creation and let the user to manually add init.py into the src field for a python target, there are cases that init.py will do something like python package initialization that may import some python files directly, if these python files also need init to be present, so the the current python package can be treated as a standard python package, this will cause a circular dependency :

package_xyz: | ----- init.py : import a, b , to do some initialization. ----- a.py : need init.py as a src/deps in the runfile tree so that this package_xyz can be marked as a standard package for things like pkg_resources to load resource files. this will be a circular dependency. ----- b.py

also i dug up bazel code over the weekend, basically that create empty init file come from here : https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/rules/python/PyBuiltins.java;l=263;drc=b5c273acbbaea9dbf6c218ec98b1310e0dde442e;bpv=1;bpt=1

which downs the road has this : https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/rules/python/PythonUtils.java;l=100;drc=b5c273acbbaea9dbf6c218ec98b1310e0dde442e

The proposal should not require too much with a new flag.