Proper setup for editable installs

[burgholzer]

Hey 👋🏼

First of all, many thanks for creating this project. Really great to see how nanobind and scikit-build-core come together. I am currently in the process of migrating some of code-bases from pybind11+setuptools to nanobind+scikit-build-core. Most of the setup was surprisingly easy (props to that). However, I am facing some peculiar problems with editable installs.

After setting up the project and performing an editable install via

git clone https://github.com/wjakob/nanobind_example.git
cd nanobind_example
python3 -m venv .venv
source .venv/bin/activate
pip install nanobind scikit-build-core[pyproject]
pip install --no-build-isolation -ve .

my IDE can't seem to find the definition for add in the nanobind_example package. And highlights it with

Cannot find reference 'add' in 'imported module nanobind_example | __init__.py' 

After some digging, I think I found out, why that might be the case. Even in editable mode, the shared nanobind_example_ext library gets installed into the site-packages directory (under the nanobind_example folder). However, since an editable install was performed, the site-packages directory does not contain the __init__.py file from the package directory. An IDE like PyCharm or CLion will notice the site-packages/nanobind_example folder and consider that for looking up definitions. Since that directory does not contain the __init__.py file, it just shows the above error message.

In my past projects using setuptools and pybind11, an editable install always led to the shared library being copied to the in-source package directory. In this case, that would be src/nanobind_example. No package directory was created in site-packages. In that fashion, IDEs were always able to pick up the proper definitions.

Long story short: It seems unintended to me that an editable install installs something to site-packages and it kind of breaks IDEs. I would have expected the editable install to install the shared library to the source tree and not to site-packages.

Tagging @henryiii since I am not quite sure whether this is something that can be resolved at the nanobind side or the scikit-build-core side.

Probably related: https://github.com/scikit-build/scikit-build-core/issues/374

---

Another side-effect that is kind of related to the above issue: Due to the use of a relative import in https://github.com/wjakob/nanobind_example/blob/1c2a1224f3de18f9840c6af1982063c5918ab654/src/nanobind_example/init.py#L1 this import can never be resolved by an IDE at the moment. I might be wrong here, but I think it would be cleaner to resorting to an absolute import:

from nanobind_example.nanobind_example_ext import add

This works both for a standard installation and in editable mode. Happy to submit a PR if you agree.

[henryiii]

A build backend should not put anything into the source directory by default. It's perfectly valid to have two virtual environments, with two different Python versions, running from one source tree. The setuptools way for doing this was broken.

IDE's are struggling with modern editable installs. The PEP that was accepted (660) doesn't provide a story for static analysis, as editable installs are very much a dynamic thing. The PEP that was rejected (662) would have supported this, but 660 was chosen instead. The problem with 660 is it basically leaves the implementation undefined, while 662 specified the implementation.

There's currently one known problem with editable installs in scikit-build-core at the moment (in the issue you linked!); we don't support importlib.resources properly. I'll be fixing that next. Maybe that will help IDEs? It depends on what they are doing and how much static vs. dynamic analysis they use.

Most other packages are moving away from simple .pth files for editable installs too, so I'm assuming IDEs are also struggling with those, like modern setuptools (at least one of it's something like 3 modes now), meson-python, and others. There might be something they are looking for that we can take advantage of, too.

I'm also fine to add another editable install mode (there's a config setting already for the mode, it just only supports one mode currently) with an in-source mode, someone just needs to write it, and in-source would need to be an opt-in mode.

Due to the use of a relative import

This is basically a namespace package - you don't have to have all files physically in the same directory, which has existed since Python 3 and IDE's should support. There may just be something we (scikit-build-core) needs to do to help IDE's.

Have you tried adding a .pyi file? I'd highly recommend that anyway, and IDE's might actually be perfectly happy with that. An IDE can't statically look into a .so anyway, so not only will it know that there is such a file, but it will be able to provide exact names and argument validation, etc.

(That's also why I'd like for support for stubgen to be a priority eventually for pybind11 and maybe nanobind)

[henryiii]

FYI, here are my stub files for boost-histogram: https://github.com/scikit-hep/boost-histogram/tree/develop/src/boost_histogram/_core - they describe boost_histogram._core, the shared lib. Based on an initial run with stubgen then lots of manual updates because pybind11 doesn't produce great output for stubgen currently.

[burgholzer]

Thanks a lot for the detailed response and the backstory on editable installs. I was loosely following these developments, but wasn't fully are of the consequences. I figured that the setuptools way was kind of broken and am really happy to get rid of it.

Have you tried adding a .pyi file? I'd highly recommend that anyway, and IDE's might actually be perfectly happy with that. An IDE can't statically look into a .so anyway, so not only will it know that there is such a file, but it will be able to provide exact names and argument validation, etc.

Just quickly tried that by adding

from __future__ import annotations

def add(a: int, b: int) -> int:
    ...

as src/nanobind_example/nanobind_example_ext.pyi and that has at least resolved that issue. Might be worth adding such a stub file to this repository. However, even with the stubfile in place, after an editable install, IDE's such as CLion and PyCharm will currently show something like with the warning message from the previous post. As stated above, I believe this is due to the current configuration creating a nanobind_example folder in site-packages and placing the shared library (and only that file) there. Should the shared library really be there for an editable install? Where, if not in the source directory would be the appropriate place to put it?

There's currently one known problem with editable installs in scikit-build-core at the moment; we don't support importlib.resources. I'll be fixing that next. Maybe that will help IDEs? It depends on what they are doing and how much static vs. dynamic analysis they use.

Might these changes yield a solution?

[burgholzer]

@henryiii any answers to the above 👆🏼?

[henryiii]

Might these changes yield a solution?

I'm not sure, but the improved version is a WIP at https://github.com/scikit-build/scikit-build-core/pull/399.

Mixing an source directory and an installed directory is rather new territory, but it's the write solution for this, I think, and other tools are moving in that direction (like meson-python), so hopefully we can get IDEs working with it too.

(Does adding the __all__ affect this?)

[burgholzer]

(Does adding the __all__ affect this?)

At least in my tests removing/adding __all__ did not influence the warning from above. I do feel that the warning really only exists because there is a package folder in the site-packages directory that contains the shared library, but nothing else.

If I change the test file to

import nanobind_example.nanobind_example_ext as m

def test_add():
    assert m.add(1, 2) == 3

everything works just fine and the IDE can resolve the extension module (because it is present in the .../site-packages/nanobind_example directory.

[tttapa]

I discuss some different ways of dealing with binary files in editable installs in the py-build-cmake documentation: https://tttapa.github.io/py-build-cmake/Editable-install.html

[burgholzer]

I discuss some different ways of dealing with binary files in editable installs in the py-build-cmake documentation: https://tttapa.github.io/py-build-cmake/Editable-install.html

Thanks for pointing this out! All of these options seem to be worth a shot. @henryiii is there any chance options like these could make it into scikit-build-core?

[wjakob]

I will close this ticket -- I think that all of these rough corners will go away once stub generation is part of the released nanobind.

[henryiii]

FYI, scikit-build-core now has two modes for editable installs, hook and inplace. Inplace has drawbacks, but it's the most "build_ext --inplace"-like method.