prefix-dev
Support cross platform / cross compiling
Problem description
I am new to pixi, and also new to the conda ecosystem. My interest in pixi is that I'm attempting to contribute to an open source project that uses pixi, CMake, C++ and Python.
I really like how pixi follows what I think of as the nodejs / node_modules pattern. I run pixi install and, thanks to the lock file, I get exactly what every other developer using pixi gets.
It occurs to me that this model would work great for cross compiling. CMake has support for cross-compiling and clang is always a cross compiler. But the hard part, or at least a hard part, has always been creating the full cross compiling environment. A big part of that is installing all the required dependencies.
One other complication is keeping "host" vs "target" straight. Anything that needs to execute needs to be compiled for the host platform, but your artifacts need to be compiled for the target platform.
It seems to me that pixi is in a good place to solve at least part of that problem. It already supports multiple platforms and multiple environments. Although, the more I think it though, the more complicated it sounds.
I guess there's a couple of asks here. I'll try to break it down. The list goes more "out there" as it progresses. Sorry about that. Please take this as me sharing my over ambitious (but unfunded) ideas, and not as making ridiculous demands.
- document the current state of cross compiling support
- it's kind of frustrating searching for that in the docs now, because it does talk about cross platform and multiple environment support, but never mentions "cross compiling", not even to say "not supported"
- support cross compiling:
- allow installing multiple platforms / non-native platforms
- this probably requires some thought. is each platform an environment? is it more of an overlay?
- for it to actually work, you need to run to host binaries for compiling and building, but link to target libraries. some headers are also target specific
- really good support for cross compiling:
- some kind of CMake integration?
- other stuff?
- remote overlays
- docker integration
- ssh integration
- emulator integrations (qemu, wine, etc)
Hi @tringenbach and welcome to the Conda ecosystem & Pixi!
Indeed, cross-compiling is a very interesting topic, and not yet widely explored with pixi itself.
As you might know, there is a "pure" build tool called conda-build and it's successor rattler-build that support cross-compilation. There are a few docs here on how this works: https://rattler.build/latest/compilers/
If one would set up the environments properly and select the correct compiler, cross compilation with pixi should be feasible. For example, to compile to linux-aarch64 you woudl want to use gcc_linux-aarch64 as a dependency and it should set everything up for cross-compiling to this target.
However, I will caution that I think the user experience for interactive development might not yet be excellent.
Maybe this can give you a few more pointers as where to look and get started.
Lastly, we are currently working on pixi build that combines rattler-build + pixi and should also hopefully support automatic cross-compilation whereever possible in the future!
Hi @wolfv - I'm also relatively new to the conda ecosystem and have been really enjoying the speed, task system, multi-language and multi-environment features of pixi!
If one would set up the environments properly and select the correct compiler, cross compilation with pixi should be feasible.
Could you possibly elaborate a little more on this? For context, I'm currently evaluating a number of tools incl. conan, spack, and pixi for managing cross-compilation development environments - not necessarily building a conda package which is why I haven't dug deep into rattler-build.
In my current experiments I'm trying to set up an environment in which I can cross compile for linux-aarch64 on an x86_64 build machine - i.e. call something like pixi run -e x-compile cmake --preset <some-x-compile-preset>
I was successful at doing what you said and depending on gcc_linux-aarch64 in a "x-compile" environment / feature. However, for other packages which don't append the platform name to their package name e.g. sdl2 the only way I've been able to get pixi to download a linux-aarch64 binary on x86 is to use the url syntax and directly depend on the linux-aarch64 package in conda forge, but transitive dependencies are still resolved for linux-64.
I also thought subdir might work but I have not had success there. e.g.
sdl2 = { version = ">=2.32,<3", channel = "conda-forge", subdir = "linux-aarch64" }
... I still get a linux-64 sdl2 conda package in my pixi.lock
To me it seems right now the biggest thing pixi is missing is a rattler-build like --target-platform or --host-platform CLI argument which would install [build-dependencies] and [host-dependencies] correctly for a specific environment. I see these are already documented parts of pixi.toml but haven't seen a way in which they're used (outside of pixi build).
I understand that this an evolving area of the project, but if you have any tips or ideas on how to download cross-platform dependencies in the current syntax that would be very helpful!
Hi @tringenbach and @ashirsch, let me illustrate how cross-compilation would work with the current CMake build backend and the cpp-sdl example:
[package]
authors = ["Bas Zalmstra <[email protected]>"]
description = "Showcases how to create a simple C++ executable with Pixi"
name = "sdl_example"
version = "0.1.0"
[package.build]
backend = { name = "pixi-build-cmake", version = "0.1.*" }
channels = [
"https://prefix.dev/pixi-build-backends",
"https://prefix.dev/conda-forge",
]
[package.host-dependencies]
# This ensures that SDL2 is available at build time.
sdl2 = ">=2.26.5,<3.0" # <- this dependency is for the _HOST_ platform e.g. `linux-aarch64`
Then one could run - on e.g. regular linux machine:
pixi build --target-platform linux-aarch64
To build to the linux-aarch64 target platform.
I'll close this now, but feel free to open new discussions/issues if you have further questions about pixi-build!
