Editable Inplace Builds Respect `build-dir`

[LeiWang1999]

Background

• We recently migrated TileLang from setuptools to scikit-build-core. With setuptools we could run python setup.py build_ext --inplace, rebuild C++ extensions in seconds, and keep artifacts out of site-packages.
• Using pip install -e . with scikit-build-core currently produces a wheel, which slows down rebuilds, and drops editable binaries into the environment’s site-packages directory, which pollutes the environment.
• Switching to mode = "inplace" avoids site-packages, but it ignores build-dir and writes artifacts directly into the project root, making the tree noisy.

What this PR changes

• Adds a new [tool.scikit-build.editable].build-dir option that is honored when editable.mode = "inplace", letting us keep a dedicated build tree (e.g. build/{wheel_tag}/editable) without resorting to site-packages or the project root.
• Updates the wheel builder to:
  • Reuse a single formatting payload for build-dir expansion.
  • Write the selected editable build location into CMake cache entries (SKBUILD_EDITABLE_MODE, SKBUILD_EDITABLE_BUILD_DIR) so CMake projects can detect the setting.
  • Avoid logging the build directory twice when we fall back to the source tree.
• Extends the dataclass model, JSON schema, and docs to describe the new knob.
• Adjusts editable-mode tests to pass the new setting and exercises the inplace path with a dedicated build directory.
• Extends the sample simplest_c CMake project to honor the new cache entries and keep its module output inside the source tree when an external build directory is used.

Remaining follow-ups

• Mirror the SKBUILD_EDITABLE_* handling in the other editable fixture projects (tests/packages/navigate_editable, tests/packages/cython_pxd_editable/pkg{1,2}, and tests/packages/importlib_editable/**/CMakeLists.txt) so every sample keeps extension artifacts in the source layout when editable.build-dir is set.
• Add a short CMake snippet to the user docs (e.g. docs/configuration/index.md) showing how projects can detect SKBUILD_EDITABLE_BUILD_DIR and route outputs correctly.
• Run the editable-focused test suite (pytest -k editable) to confirm there are no regressions.

[LecrisUT]

Thanks, I had a branch working on this on the side, but if someone else can work on this, it will help get this in much faster. I will give a quick review to discuss the design aspect that we would want to have for this.

[LeiWang1999]

I can succesfully reach my goal via pip install -e . --no-build-isolation with this commit, this pr is ready for review.

[LecrisUT]

I can succesfully reach my goal via pip install -e . --no-build-isolation with this commit, this pr is ready for review.

But this is not what we would want to have. inplace mode is explicitly for the purpose of having in-source builds where the build artifacts are alongside the source files, e.g. if you have resource files in the source folder that you access via __file__ location. What you are describing there is what redirect mode already mostly does but with a few steps that is making it a bit slow and which we would want to improve on.

[LeiWang1999]

@LecrisUT I see your point. Would the plan be to extend redirect with a switch that skips the wheel staging altogether? That sounds like a much larger patch. And, I can not understand the usage of inplace actually.

[LeiWang1999]

Anyway, I’m happy to rework the implementation along the lines you’ve suggested—my main goal is just to land an update quickly so we can get our development workflow unblocked.

[LecrisUT]

@LecrisUT I see your point. Would the plan be to extend redirect with a switch that skips the wheel staging altogether?

Yes, and indeed it is not a quick drive-by patch for this, but we can assist you all-throughout the process if anyone wants to pick up the mantle for that.

my main goal is just to land an update quickly so we can get our development workflow unblocked.

What is wrong with redirect mode? Specifically with rebuild = false. In that case the intended workflow is something like this:

• pip install -e . --config-settings=build-dir=build (already defaults are redirect mode and rebuild = false)
• manually do cmake --build and cmake --install whenever meaningful changes to the files are made

It will reuse the previous build directory and if all of the cmake dependencies are picked from the system, a CMake re-configure step should not be done (if it is, please try to hunt down what causes it, it's mostly a project configuration issue)

[LeiWang1999]

@LecrisUT Thanks, that works for me as long as we enable --build-no-isolation. my folks use scikit-build-core usually just do pip install -e . lol.

[LecrisUT]

@LecrisUT Thanks, that works for me as long as we enable --build-no-isolation

This issue pops up regularly in the issues, most simple example of this is in usage of pybind11. In principle, if you have a dependency installed on the system that find_package is picking up or you installed it in your own venv and use --build-no-isolation the issue of re-configure should go away. We should make the interaction with editable mode smoother, but we don't know yet how we should do it. One way could be to have a hook to try and do find_package so that system packages (which do not vanish under build isolation) are picked up and preferred. We mostly need feedback and discussions on this to find what would be preferred.

[LeiWang1999]

But one thing I have to complain about is that the editable build (pip install -e .) copies the build artifacts into site-packages, which is unreasonable and pollutes the environment. For example, I encountered this:

> pip install -e . --no-build-isolation
> ls somepath/python3.12/site-packages/tilelang
3rdparty  lib

As a result, sometimes when I run a program, it works fine:

> python
Python 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tilelang
>>> tilelang.__path__
['local_dir/tilelang/tilelang']

But other times, it fails to locate the right path — even when I explicitly set PYTHONPATH:

# test_files.py
import tilelang
print(tilelang.__path__)
from tilelang import language as T

> python test_files.py
> _NamespacePath(['somepath/python3.12/site-packages/tilelang'])

Then it throws:

ImportError: cannot import name 'language' from 'tilelang' (unknown location)

because that directory only contains lib and 3rdparty.

[oraluben]

We do sometimes want pure pth solution, i.e. to have full src and build artifacts in source folder. Meantime, we also want to set 'build-dir' when using editable.mode = "inplace" to not mess up the current source dir (to build in a seperate in-tree build dir, which is also a common practice for cmake-based project). Given that, I think there's no harm to respect build-dir for inplace mode.

[LecrisUT]

Meantime, we also want to set 'build-dir' when using editable.mode = "inplace" to not mess up the current source dir (to build in a seperate in-tree build dir, which is also a common practice for cmake-based project).

We are well aware of the CMake good practices and we often discuss those in the issues. The inplace mode is not a recommended method for developers, and its primary usage is when the project hard-codes relative paths to resource files like

from pathlib import Path
DIR = Path(__file__).parent
SOME_DATA = DIR / "resources/data.csv"

The modern approach around this is to use importlib.resources. However we still need to support this older approach in order to ease the developer's burden for doing such transition.

Given that, I think there's no harm to respect build-dir for inplace mode.

There is, because the inplace mode is explicitly designed to be in-source build mode. E.g. the SOME_DATA in the example above could be a file generated by CMake using configure_file, add_custom_command, etc. which are later installed in the CMake to the package folder. In the inplace mode we are skipping the installation and rely on the fact that the build directory is mirrored with the source directory.

If you do not use such constructs, do not set the editable mode to inplace. Instead use the redirect which is the default.

But one thing I have to complain about is that the editable build (pip install -e .) copies the build artifacts into site-packages, which is unreasonable and pollutes the environment. For example, I encountered this:

Yes, this is the unfortunate limitation with the redirect mode. This mode is meant to use the cmake --install in order to know which files need to be linked in the _<module>_editable.py file that we place in the site-packages. This is also the method that we would want to change with introducing a new editable mode.

If you are interested in the details

Basically we have the file-api interface from CMake which allows us to interrogate all of the CMake install commands and we can make a list of the locations in build-dir and the intended locations in the install path. And then we can do the same thing that we do to handle the source files but pointing to the build-dir https://github.com/scikit-build/scikit-build-core/blob/3f371fff3d0dd424811ef73f58d88389263d79db/src/scikit_build_core/resources/_editable_redirect.py#L232

But other times, it fails to locate the right path — even when I explicitly set PYTHONPATH:

The resolutions of __path__ is something rather complex. Ideally do not rely on it, instead use importlib.resources which can account for complex namespace installations, e.g. editable installs from different projects that share the same package, e.g. some projects install into the common sphinx.contrib as a form of simple plugin mechanism. When using editable installs you are likely to end up in a similar situation regardless of build backend or editable mode chosen.

We are aware that __path__ and other related methods do not always work as expected and those are tracked in https://github.com/scikit-build/scikit-build-core/pull/808, but we really need more help with that.

[LeiWang1999]

The new mode is super important for us.

[LecrisUT]

The new mode is super important for us.

I will try to carve some time this weekend to publish the WIP that I have. Please check back on me, if I don't do any updates.

[henryiii]

I think the name of the new mode could be editable.mode = "build-dir".

[LecrisUT]

@LeiWang1999 @oraluben Please follow #1170 for further progress on this. I will try to note the caveats of this as we pick them up, but a major caveat is #808 and getting importlib.resources to work properly.