STEllAR-GROUP
Add HPX Conda package to conda-forge
Most of the users of our software use the Conda package manager to install it. Future versions of our software depend on HPX. HPX is not available as a Conda package, so building a Conda package for our software involves building HPX first itself as well. This works, but is not ideal. It takes time, complicates matters, and will likely result in crashes once other projects depending on HPX start to do the same thing.
It would be great if HPX, like Boost and all the other libraries we use is available as a Conda package. This would help us, and any other Conda package depending on HPX.
- Does the HPX team know of anybody working on a Conda package for HPX?
- If not, is the HPX team willing to support someone who will have a go at this?
Does it really make sense to create a package for a Python package environment that will build a C++ package? FWIW, HPX is available as a prebuilt package through the Fedora package management, spack, vcpkg, and others.
OTOH, if you plan to create a Conda package, please let us know if we can help in any way.
Conda is commonly used with python. However, its goal is actually to be a full environment management solution. I find the idea of packaging HPX into conda quite intriguing. I agree with hkaiser on it not making much sense.
Another question that needs to be asked. Which configuration will be compiled ? HPX without MPI? HPX with APEX? HPX with tcmalloc?
I am willing to give it a shot next month. But only if it makes sense for HPX
I am willing to give it a shot next month. But only if it makes sense for HPX
I'm not opposed to the idea if somebody would be willing to contribute such a package and to maintain it in the foreseeable future. As said, we'd be happy to support such an effort.
Great to hear that the HPX team is, in principle, willing to support an HPX Conda package!
A bit more background:
Our software (called LUE) is mostly written in C++, but provides end-users with a Python API. To our users it is a Python package. That is why it is convenient for our users to distribute LUE as a Conda package. They can then install it using conda install lue and develop Python scripts that use LUE as well as other Python packages.
Conda packages must be built within the Conda build environment. Luckily, almost all libraries and tools needed to compile C++ software are available within the Conda build environment.
BTW, once HPX is a Conda package, it is regularly rebuilt on various platforms. Factors that trigger a rebuild are for example new versions of libraries HPX depends on, new versions of compilers, etc. Potentially, the HPX build scripts or sources may have to be updated because of this. This makes HPX more portable than it already is, but probably requires the assistance of the HPX team.
With respect to the configuration: my preference would be to configure HPX with common-sense defaults and good performance on each platform. Conda packages are mostly meant for end-users of Python APIs, not for C++ software developers. I think most / all Conda package users are using shared memory systems. tcmalloc is available in the Conda build environment of Linux and macOS. Users with special wishes can build our software, including HPX, themselves.
Hi @John98Zakaria , are you still planning to work on packaging HPX as a Conda package? If not, we will probably have a go at it. But if you do, then we are happy to wait a bit more and keep working on other stuff ;-)
@kordejong I just got my bachelor's degree and need a break from using the PC. You can give it a shot yourself. Sorry for not informing you earlier
@hkaiser I will start work on adding an HPX Conda package to conda-forge. My colleague @oliverschmitz and I will act as maintainers of the package, but it would be good to also add an HPX expert to the list of maintainers. Oliver and I will handle the packaging work while the HPX expert would be responsible for providing patches for the HPX source code, should that be required. I don't expect there will be many of those because HPX is portable software and version 1.8.1 is already part of a Conda package of our own software. In the future patches might be needed when new versions of HPX are released, or when new versions of packages HPX depends on (Boost, gperftools, ...) are released. We are planing to support HPX packages for osx-arm64, osx-64 (Intel), linux-64, and win-64. If you agree with this, can you provide us with the Github ID of an HPX expert who is willing to be co-maintainer of the package? This person will receive e-mails related to the packaging of HPX, most of which he/she can ignore because they are not related to the HPX sources themself. We can explicitly ping this person when HPX-specific expertise is required.
If you agree with this, can you provide us with the Github ID of an HPX expert who is willing to be co-maintainer of the package?
For the time being, please add myself. Thanks!
I have configured a simple HPX Conda package and submitted it for inclusion in Conda-Forge. I hope the package is correct and they will accept it. To keep things simple, HPX is currently built without support for networking. It is possible to support package variants, for example nompi, mpich, openmpi. The HDF5 Conda package does that, for example. This involves more configuration, which I plan to do later.
@hkaiser: FYI, the PR is here: https://github.com/conda-forge/staged-recipes/pull/20916 The interesting stuff is in recipes/hpx. The patch subdirectory contains a few patches to allow HPX-1.8.1 to build on all platforms. It would be good if you could review these. They may be relevant to include in the current HPX sources, if that is still needed.
The package can be updated at any time, also for a specific version of HPX for which a Conda package has been released. The goal now is to get a simple initial version of the package accepted into Conda-Forge, and improve on it over time.
The first version of the package is ready for use. The feedstock repo and information about the package are here:
- https://github.com/conda-forge/hpx-feedstock
- https://anaconda.org/conda-forge/hpx
I have created some tickets in the feedstock repo that, once fixed, will make using the package more convenient.
@John98Zakaria Let me know if you want to help maintain and improve the Conda package. I can add you to the list of maintainers. @hkaiser I think this issue can be closed now.