Cannot mix native/implicit and pkg_resources namespace at all

[mauritsvanrees]

With pip, as far as I see, the following is true:

• Within one namespace you cannot use a checkout (development/editable install) of one package with old-style pkg_resources/setuptools namespaces and another package with new-style native/implicit pep-420 namespaces.
• But it is fine to have one namespace style in the checkouts and another (or both) in normal (non-editable) installs.

For Buildout this is sadly not true. You cannot mix two styles of namespaces at all. If you have an editable install with native namespace you cannot have a normal install with pkg_resources namespace, and the other way around.

For clarity, it is fine to have for example all zope.* packages with native namespaces and all `plone.* packages with pkg_resources namespaces.

It is a bit hard to explain how to try it out, but let me try.

mkdir foo
cd foo
python3.13 -mvenv .
bin/pip install -U pip setuptools zc.buildout wheel build -c https://dist.plone.org/release/6.1.0/constraints.txt

Create a buildout.cfg here:

[buildout]
extensions = mr.developer
extends = https://dist.plone.org/release/6.1.0/versions.cfg
prefer-final = true
find-links = ./wheels
parts =
    zopepy
auto-checkout +=
#    five.intid
#    five.customerize
#    five.localsitemanager

[versions]
# don't care about differing versions for these:
pip =
setuptools =
zc.buildout =
# five.customerize = > 3.5
# five.intid = > 2.5
# five.localsitemanager = > 4.5

[zopepy]
recipe = zc.recipe.egg
eggs =
    five.localsitemanager
    five.customerize
    five.intid
interpreter = zopepy

[sources]
five.customerize      = git [email protected]:zopefoundation/five.customerize.git branch=pep-420-native-namespace
five.intid            = git [email protected]:plone/five.intid branch=pep-420-native-namespace
five.localsitemanager = git [email protected]:zopefoundation/five.localsitemanager branch=pep-420-native-namespace

Run bin/buildout and you will get a bin/zopepy script with three five.* packages. You get the versions pinned by Plone 6.1.0, which are pkg_resources namespaces. This does not use checkouts yet. All is well.

@icemac has made pull requests for three packages in the five namespace to port them to native namespaces. Two are in the zopefoundation org and one in plone.

If you enable these all three in the auto-checkouts list and run bin/buildout again, all is well: all have the native namespaces style.

But when you use only one or two checkouts, it means you have two namespace styles, and things break: you can import some of the packages, but not all three.

You can create binary wheels for all three packages and put them in a new wheels directory, which is in the find-links. For example:

mkdir wheels
cd src/five.localsitemanager
../../bin/python -m build .
mv dist/five.localsitemanager-5.0.dev0-py3-none-any.whl ../../wheels/

Then edit the versions part in buildout.cfg and uncomment some lines there so you use one or more of these wheels, and still a checkout of the others, and it will be fine again: import of all packages works. So once all source checkouts or final packages of one namespace are native namespaces, all is well again.

Conclusion: if you use buildout, all packages within one namespace must be switched to native namespaces at the same time.

This can probably be fixed. But I have no idea if this takes two hours or twenty days. Currently I am not inclined to take this on. :-/

[mauritsvanrees]

Small update, because I think one part is not clear in the above. Edit the buildout config to have these version pins:

five.customerize = 4.0
five.intid = 2.0.0
five.localsitemanager = 5.0

The five.intid pin is the same as in Plone 6.1.0, so it has pkg_resource namespaces. The other two are the latest releases and they have native namespaces.

Result after running bin/buildout: only five.intid can be imported, the others not. So you also cannot mix two namespace-styles for final releases.

[mauritsvanrees]

@davisagli came up with an idea:

I see buildout has some special handling for namespace packages here, to add __init__.py files: https://github.com/buildout/buildout/blob/4.1.4/src/zc/buildout/easy_install.py#L1803 So maybe we could just add a native-namespace option to buildout to tell it to NOT do that...

I think this could work. At least it is worth a try. It could spectacularly fail of course, but we don't know until we try. :-) Maybe we would need to remove namespace_packages.txt from the EGG-INFO directory as well.

I think eggs generated in this way should be stored in a separate place. Otherwise a buildout config that does not use this option, would still get the "new" egg and this would fail. So you would get something like this:

eggs
eggs/cp313
eggs/force-native
eggs/force-native/cp313

With force-native = true in the buildout config, the eggs would end up in eggs/force-native. With additionally abi-tag-eggs = true, the eggs would end up in eggs/force-native/cp313 (depending on Python version).

Only eggs with pkg_resources style namespaces would end up in the new location. Eggs that already use native namespaces, or that don't have a namespace at all, would end up at the original locations, because there is nothing to change there.

Maybe the same should be done for pkgutil style namespace packages, but we could wait until someone asks for it. I am not aware of any packages that use this style.

[davisagli]

@mauritsvanrees Good point, I hadn't considered that this gets a bit more complicated because eggs can be shared by different buildouts, but that makes sense.

[mauritsvanrees]

I have added an experiment that looks promising. See PR #697.

[davisagli]

I made an experimental tool which tries to make pkg_resources-style namespace packages compatible with the other styles: https://pypi.org/project/horse-with-no-namespace/

[gotcha]

@mauritsvanrees released 5.0.0a1 which partially solves this issue. Check the README where details are provided.