Editable Install with cython extension and shared library for ctypes

[AndrewAnnex]

I have a python project that includes a cython extension linked against a shared library I also interface to using ctypes in the same project, and I am trying to enable editable installs to allow for the conventional build_ext --inplace workflow that I've used to develop the package in the past, but I am running into odd behavior with the replace and inplace editable installation modes and likely my inexperience with cmake is making this difficult to reason about. I am aware of the number of related issues with editable installs, however after reading through a number of them I am still stuck on this.

It could be I am doing something nonstandard in my cmakelists.txt file, but so far what I have works for building installable wheels where all tests pass in my CI/CD with cibuildwheel and pipx.

link to my CMakeLists.txt link to my pyproject.toml

In the end, I need both the cython extension (for my mac cyice.cpython-313-darwin.so) and the shared library (libcspice.dylib) to be installed in my project root folder rather than just site-packages. The expected paths for each would then be src/spiceypy/cyice/cyice.cpython-313-darwin.so and src/spiceypy/utils/libcspice.dylib

with redirect and pip install --no-build-isolation -ve .

Logs:

  *** Installing project into wheel...
  2025-09-04 09:37:08,639 - scikit_build_core - INFO - RUN: /opt/homebrew/bin/cmake --install build --prefix /var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/tmpb0blklrg/wheel/platlib --strip
  -- Install configuration: "Release"
....
  -- Installing: /var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/tmpb0blklrg/wheel/platlib/spiceypy/utils/cmake/cspice/cspiceTargets.cmake
  -- Installing: /var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/tmpb0blklrg/wheel/platlib/spiceypy/utils/cmake/cspice/cspiceTargets-release.cmake
  -- Installing: /var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/tmpb0blklrg/wheel/platlib/spiceypy/utils/cmake/cspice/cspiceConfig.cmake
  -- Installing: /var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/tmpb0blklrg/wheel/platlib/spiceypy/utils/cmake/cspice/cspiceConfigVersion.cmake
  -- Installing: /var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/tmpb0blklrg/wheel/platlib/spiceypy/utils/libcspice.dylib
  -- Installing: /var/folders/6r/0_ck04691rx1vj2x5knvq8s00000gn/T/tmpb0blklrg/wheel/platlib/spiceypy/cyice/cyice.cpython-313-darwin.so

but both the shared library and the cython extension are only found in my site-packages folder:

ls /Users/andrew/mambaforge/envs/spiceypydev_313/lib/python3.13/site-packages/spiceypy/utils                                        
libcspice.dylib
ls /Users/andrew/mambaforge/envs/spiceypydev_313/lib/python3.13/site-packages/spiceypy/cyice/                                      
cyice.cpython-313-darwin.so

with inplace and pip install --no-build-isolation -ve .

as build-dir is now ignored, a lot of build folder stuff is made in my project root folder, including the cython extension which is sitting at the root path instead of inside my src directory in src/spiceypy/cyice/ where i'd expect it to be. The shared library remains in _deps and doesn't get copied anywhere else.

  [2231/2235] Linking C shared library _deps/cspice-build/libcspice.dylib
  [2232/2235] Linking C executable _deps/cspice-build/test_cspice
  [2233/2235] Cythonizing /Users/andrew/PycharmProjects/SpiceyPy/src/spiceypy/cyice/cyice.pyx to /Users/andrew/PycharmProjects/SpiceyPy/spiceypy/cyice/cyice.c
  [2234/2235] Building C object CMakeFiles/cyice.dir/spiceypy/cyice/cyice.c.o
  [2235/2235] Linking C shared module cyice.cpython-313-darwin.so
  *** Making editable...

I am wondering if the cyice.dir is a clue here, as this causes a copy of cyice.c to be made outside of the src directory making a new folder at spiceypy/cyice relative to the project root.

[henryiii]

If you want inplace to work, then you should move the CMake code to build to src/spiceypy/cyice/CMakeLists.txt. With an inplace build, you are not installing, so items remain where they are defined.

If you want redirect to work, why can't you put the files into site-packages? That's what you have to do when shipping non-editably, so why would you want it to be different in editable mode? If you are adding binary files to the source directory, you won't be able to make more than one build, for example. I suppose you could install into the source directory (unless there's a check for that in CMake), but I'd carefully guard it for the specific case where you have to have it, since it's not standard and will trip others up.

[AndrewAnnex]

@henryiii I hadn't considered putting the cmake code in that folder before, but I can see a bit about why that could make sense. I will look into trying that, but again really my cmake is installing both the c library and the cython extension so I am not sure that this totally makes sense to do just yet to me.

My truly naive understanding of editable installs in python, and how it has been working before with cython in-place compilation, was that my src/spiceypy would be symbolically linked to site-packages/spiceypy (or something close to that). When compiling in-place, the resulting shared libraries would remain in src/spiceypy/cyice/cyice.so and such. If I were to run pytest in the parent directory to src, that is the root folder of the project, pytest would correctly see the .so files in the src subfolders rather than expecting them to be in site-packages directly.

I also don't totally follow what the redirect mode is attempting to do versus inplace, so maybe that's part of my confusion here, but I do know what behavior I want, which may just be the inplace mode precisely.

I'll need to think about this more clearly...