is there a procedure to install xtensor-python via standard pip on windows ?

[stonebig]

maybe it's impossible...

[SylvainCorlay]

At the moment we have not provided one. We could do this via the install_headers directive.

My idea on this was to create an xtensor-pip package that would vendor xtensor and xtensor-python and install the headers the install_headers pip directive.

The drawback is that the headers are then (typically) installed under the python include directory - which is genrally different from the general include directory of the prefix (include vs include/pythonX.Y).

Maybe we could do this with python wheels' data files instead. If you have a suggestion, that would be great?

[SylvainCorlay]

cc @ghisvail .

If there is a reliable means with python wheels to install into the prefix include directory, that might be the solution, and we could simply add a setup.py to the repository.

[alexandrebrilhante]

Any update on this?

[PerretB]

One could do it in pybind11 or numpy style. First, define header files as package data : they will be copied in the package directory during install. Then define a method get_include that returns the path to the local include files, see eg. get_include in pybind11.

Of course this does not follow the classical system way to do it, but it is indeed pretty convenient and OS independent in a python context.

[StefanUlbrich]

Hi, I created a pull request where I copied the respective files from pybind11. Not finished yet but works for me already: #236

[StefanUlbrich]

I drafted a meta package that provides the headers.

https://github.com/StefanUlbrich/xtensor-cpp

Maybe this is a useful extension to the xtensor-stack?

[JohanMabille]

Hi @StefanUlbrich ,

Thank you for tackling this! I'm not sure yet, but I wonder if building the wheel with scikit-build and CMake could simplify the process:

• the wheel can be done in a dedicated repo (no need to add specific packaging stuff here)
• all the files to install would be get from CMake, no need to duplicate.
• we could use CMake external projects mechanism and avoid relying on git submodules.

@SylvainCorlay any thought on this?

[StefanUlbrich]

@JohanMabille what about conan? most of the xtensor stack is already there

[JohanMabille]

I am not sure if these packages are still maintained. Anyway this is orthogonal to pip, right? I mean, having a package for conan is independent from having / building a wheel for pip, or is there some bridge that makes it easy to have one when we already have the other?

[StefanUlbrich]

@JohanMabille The packages are still maintained but you are right. My idea was to have a simple package that can be installed and a numeric extension can be set up with minimal effort using setuptools. CMake is of course standard but might be overkill for those not familiar with it–I personally never became comfortable with it when I met it 10yrs ago. In addition, some projects might require lapack and blas installation and static linking (xtensor-blas). Do you think a cookie cutter would be more adequate? I am more than willing to contribute to an "official" endorsed "starter kit"

[StefanUlbrich]

Maybe this thread is a good way to start?

[JohanMabille]

CMake is of course standard but might be overkill for those not familiar with it

I totally agree, especially for people coming from Python and not having a lot of experience with C++ projects. What I meant is that we (developers and package maintainers) should leverage on existing CMake to build packages in order to make the process easier to maintain (no need to duplicate list of files, build flags, etc).

Do you think a cookie cutter would be more adequate?

We already provide a [cookie cutter] for authoring python extensions based on xtensor, but it might be broken (I haven't tried it for a while) and anyway it deserves some love ;). I think this (cookie cutter) and a wheel are complementary since the cookie cutter does not vendor the dependencies.

I am more than willing to contribute to an "official" endorsed "starter kit"

Thanks that is much appreciated! We have already experienced with scikit-build for building a more complicated stack. The process should be simpler for xtensor since it's a header-only library an it as fewer dependencies.

We split the R-bindings and Julia bindings in two packages, one pure C++ package, and one that vendors the C++ package and its dependencies (see https://github.com/xtensor-stack/Xtensor.R and https://github.com/xtensor-stack/xtensor-r for instance). As per in person conversation with @SylvainCorlay, we should do the same with xtensor-python: keep the headers and pure C++ stuff in xtensor-python and move the Python-related stuff in a new repo xtensor.py, that would vendor the dependencies and build the wheel based on scikit-build

Let us know what you prefer, you can create this xtensor.py repo and iterate on it until you have soething that works, then transfer it to the xtensor-stack organization, or we can create it directly here. In both cases I would be super happy to help with CMake and scikit-build.

[SylvainCorlay]

We already provide a [cookie cutter] for authoring python extensions based on xtensor, but it might be broken (I haven't tried it for a while) and anyway it deserves some love ;)

CI is set up for the cookiecutters. We should definitely give it a run.

We should do the same with xtensor-python: keep the headers and pure C++ stuff in xtensor-python and move the Python-related stuff in a new repo xtensor.py, that would vendor the dependencies and build the wheel based on scikit-build.

👍 we have done this for both Julia and R.

[Coderx7]

Hey everyone, Any update on this?

[JohanMabille]

We are swamped with a lot of other tasks and don't have the time to implement it for now. We can provide some guidance if you want to tackle this.

[StefanUlbrich]

Just switched jobs. I'd love to continue within the next few weeks.

[benbovy]

That would be great to have, so that nowadays we could just use scikit-build-core with:

• a pyproject.toml

[build-system]
requires = ["scikit-build-core", "pybind11", "xtensor-python"]
build-backend = "scikit_build_core.build"

[project]
name = "my_python_extension"
version = "0.0.1"
dependencies = ["numpy"]

• a CMakeLists.txt

And then python -m pip install .

I'm already using scikit-build-core with xtensor(-python) installed via conda/mamba, which works well although it is slightly less convenient (I need to run python -m pip install . --no-build-isolation).

[benbovy]

I did some experiment here: https://github.com/benbovy/xtensor-python-wheel

I used scikit-build-core together with simple pyproject.toml files and CMake's FetchContent to create wheels for the xtensor projects. It is actually very straightforward! The repository also contains a small example that is built from the installed wheels in a github action.

Caveats:

• I've created separate projects / wheels for xtl, xtensor and xtensor-python. I haven't figured out how to customize the install destination with CMake and FetchContent... It is probably possible via ExternalProject_Add but it is a bit more work.
• The created wheels are platform specific, e.g., xtensor_python-0.26.1-py3-none-manylinux_2_35_x86_64.whl whereas we want something like xtensor_python-0.26.1-py3-none-any.whl. Not sure this is supported by scikit-build-core, but it would actually be useful for header-only (non-python) libraries supporting cmake.
• In the example project's CMakeLists.txt, we need to add find_package() for each of the xtensor projects (configured targets do not work)