grass icon indicating copy to clipboard operation
grass copied to clipboard

[WIP] Conda Recipe for Linux

Open HuidaeCho opened this issue 11 months ago • 17 comments

https://github.com/HuidaeCho/grass-conda

GRASS Conda Packaging

  1. Install Miniconda
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod a+x Miniconda3-latest-Linux-x86_64.sh
./Miniconda3-latest-Linux-x86_64.sh -b -u -p ~/opt/miniconda
~/opt/miniconda/bin/conda init
. ~/.bashrc
  1. Setup Conda for GRASS build and packaging
conda config --add channels conda-forge
conda config --set channel_priority strict
conda create -n grass-conda conda-build git
conda activate grass-conda
  1. Clone this repository
git clone https://github.com/HuidaeCho/grass-conda.git
  1. Build and package GRASS
cd grass-conda
conda-build recipe
  1. Find a new GRASS package
ls -al ~/opt/miniconda/envs/grass-conda/conda-bld/linux-64/grass-1.0.0-h3fd9d12_0.conda
  1. Create a new test Conda environment and install it
conda create -n grass-conda-test
conda activate grass-conda-test
conda install grass -c ~/opt/miniconda/envs/grass-conda/conda-bld
  1. See if GRASS is installed
ls -al ~/opt/miniconda/envs/grass-conda-test/bin/grass

HuidaeCho avatar Feb 07 '25 20:02 HuidaeCho

https://github.com/OSGeo/grass/pull/3621

CMake & Conda TODOs:

  • [ ] FHS for Python packages as well; Need to check @nilason's previous work
  • [x] Fixing stdout/stderr on Windows
  • [x] Complete Conda for Linux without FHS for now

HuidaeCho avatar Feb 07 '25 20:02 HuidaeCho

I got mixed results here. After successful execution of the commands above, I get:

/root/opt/miniconda/envs/grass-conda-test/bin/grass --help
Traceback (most recent call last):
  File "/root/opt/miniconda/envs/grass-conda-test/bin/grass", line 2473, in <module>
    main()
    ~~~~^^
  File "/root/opt/miniconda/envs/grass-conda-test/bin/grass", line 2118, in main
    find_grass_python_package()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/root/opt/miniconda/envs/grass-conda-test/bin/grass", line 2106, in find_grass_python_package
    raise RuntimeError(msg)
RuntimeError: The grass Python package is missing. Is the installation of GRASS complete?

The setting PYTHONPATH doesn't help:

PYTHONPATH=/root/opt/miniconda/envs/grass-conda-test/lib/grass85/etc/python/ /root/opt/miniconda/envs/grass-conda-test/bin/grass --help

...although executing python command and import with PYTHONPATH works:

PYTHONPATH=/root/opt/miniconda/envs/grass-conda-test/lib/grass85/etc/python/ python
>>> import grass.script as gs

...but it fails later:

>>> gs.create_project("test")
/root/opt/miniconda/envs/grass-conda-test/lib/grass85/bin/g.gisenv: error while loading shared libraries: libgrass_gis.so.8: cannot open shared object file: No such file or directory
/root/opt/miniconda/envs/grass-conda-test/lib/grass85/bin/g.message: error while loading shared libraries: libgrass_gis.so.8: cannot open shared object file: No such file or directory

...suggesting that the automated env setup mechanism can't find /root/opt/miniconda/envs/grass-conda-test/lib/grass85/lib/libgrass_gis.so.8.

I can look at find_grass_python_package later. FHS would obviously help at least with the Python part.

wenzeslaus avatar Sep 19 '25 16:09 wenzeslaus

I suspect the path on the build machine is hardcoded in that file like on Windows/OSGeo4W since July 7 https://github.com/OSGeo/grass/issues/6290

echoix avatar Sep 19 '25 17:09 echoix

Indeed, the build machine (build time) path is the one which is used as GRASS_PYDIR, set to the following in my case:

/root/opt/miniconda/envs/grass-conda/conda-bld/grass_1758295620319/work/build/output/lib/grass85/etc/python

obtained by:

grep os.path.normpath /root/opt/miniconda/envs/grass-conda-test/bin/grass

wenzeslaus avatar Sep 22 '25 21:09 wenzeslaus

With further investigation, I see that grass.py requires GRASS_PYDIR to be right, but does not take advantage of PYTHONPATH or otherwise correctly set up grass package (namely if FHS system puts with other Python packages). I also found that the build for the package did not replace build variables in the setup.py file under python/grass/script.

I opened two PRs:

  • #6391 removes the need to fix the build variable substitution
  • #6393 takes advantage of standard Python setups before trying GRASS_PYDIR (so here it helps to work around the problem, but does not fix anything since the conda package is not currently using FHS)

wenzeslaus avatar Sep 22 '25 21:09 wenzeslaus

Please test the latest https://github.com/HuidaeCho/grass-conda. It builds and installs the non-FHS version in bin/grass and opt/grass on Linux. Running grass works for me.

HuidaeCho avatar Sep 26 '25 22:09 HuidaeCho

The compilation now fails for me with:

mv: cannot stat '/root/opt/miniconda/envs/grass-conda/conda-bld/grass_1759321665374/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/lib64': No such file or directory

Details: https://github.com/HuidaeCho/grass-conda/issues/6

wenzeslaus avatar Oct 01 '25 12:10 wenzeslaus

I tried the previous version (the one I tested before, that is https://github.com/HuidaeCho/grass-conda/commit/d6c353ef09ab06b9b4a4640b33ca0f41718237fa) and that builds as before and the changes from #6391 and #6393 helped some with the package (can be imported and run commands):

PYTHONPATH=/root/opt/miniconda/envs/grass-conda-test/lib/grass85/etc/python/ python                             
>>> import grass.script as gs
>>> gs.create_project("test")
>>> gs.create_project("test2", epsg=3358)
>>> gs.setup.init("test2")
<grass.script.setup.SessionHandle object at 0x749cc2b5c1a0>
>>> from grass.tools import Tools
>>> Tools().g_proj(flags="p")
...

...and with the executable package interface (same setup as the package):

PYTHONPATH=/root/opt/miniconda/envs/grass-conda-test/lib/grass85/etc/python/ python -m grass.app run g.region -p

...and with the main executable (no PYTHONPATH needed):

grass --help

I'm testing in a fresh Docker Ubuntu container (docker run --network host -it ubuntu:24.04).

As a side note, because that might be already fixed in the new version, these still fail:

$ grass --tmp-project XY --exec g.proj -p
FileNotFoundError: [Errno 2] No such file or directory: '/root/opt/miniconda/envs/grass-conda/conda-bld/grass_1759322494942/work/build/output/lib/grass85/etc/lock'
$ grass --config python-path
/root/opt/miniconda/envs/grass-conda/conda-bld/grass_1759322494942/work/build/output/lib/grass85/etc/python

wenzeslaus avatar Oct 01 '25 13:10 wenzeslaus

With (recent) a92a3a018ace6833bc2f13cf0031ae2178d8e5ea (#6469) on main and (older) https://github.com/HuidaeCho/grass-conda/commit/d6c353ef09ab06b9b4a4640b33ca0f41718237fa, the following works:

grass --tmp-project XY --exec g.proj -p
grass --tmp-project EPSG:3358 --exec g.proj -p
grass run --crs XY g.proj -p
grass run --crs EPSG:3358 g.proj -p

Meaning the main GRASS executable is on path and it can find all its files (because of fallbacks implemented in #6391, #6393, and #6469).

The executable now also correctly returns path to Python packages:

$ grass --config python-path
/root/opt/miniconda/envs/grass-conda-test/lib/grass85/etc/python

So that can be used to set up Python path (PYTHONPATH or sys.path) without hardcoding it:

PYTHONPATH="$(grass --config python-path):$PYTHONPATH" python
import grass.script as gs
from grass.tools import Tools
gs.create_project("test1", crs="XY")
gs.create_project("test2", crs="EPSG:3358")
gs.setup.init("test2")
Tools().g_proj(flags="p")

wenzeslaus avatar Oct 10 '25 18:10 wenzeslaus

PR for conda-forge: https://github.com/conda-forge/staged-recipes/pull/31234

HuidaeCho avatar Oct 16 '25 19:10 HuidaeCho

@HuidaeCho

I’ve submitted a PR to publish the GRASS GIS conda package for Linux, supporting Python 3.10–3.13:

🔗 PR: conda-forge/staged-recipes#31234 🔗 Source branch: dudaka/staged-recipes/tree/grass-linux-8.4.1 🔗 CI runs: Pipeline status & logs

All required checks have passed, and the CI pipeline successfully built the package.

Fixing OverLinkingError

The initial build failed during conda-build post-processing with an OverLinkingError, caused by several binaries linking to DSOs not declared in requirements.run:

lib/libgdal.so.37   → provided by libgdal-core
lib/libpdalcpp.so.19 → provided by libpdal-core
lib/libgomp.so.1     → provided by libgomp

Representative errors:

ERROR (grass,grass84/bin/g.proj): Needed DSO lib/libgdal.so.37 … not in reqs/run
ERROR (grass,grass84/bin/r.in.pdal): Needed DSO lib/libpdalcpp.so.19 … not in reqs/run
ERROR (grass,grass84/lib/libgrass_gpde.8.4.so): Needed DSO lib/libgomp.so.1 … not in reqs/run

Fix applied (in meta.yamlrequirements.run):

  • ibgdal-core (for lib/libgdal.so.37)
  • libpdal-core (for lib/libpdalcpp.so.19)
  • libgomp (for lib/libgomp.so.1, OpenMP support)

These were added explicitly since the recipe currently ignores run_exports for GDAL/PDAL. Alternatively, ignore_run_exports could be adjusted to let their run_exports automatically include these split libraries.

Testing the recipe locally

The build-locally.py script used for local testing is available in the conda-forge/staged-recipes repository.

Method 1 — Using build-locally.py (recommended)

# From the root of staged-recipes
python build-locally.py          # choose linux_64.yaml interactively
# or directly:
python build-locally.py linux_64

  • Runs inside a Docker container that replicates conda-forge’s CI environment.
  • Build artifacts are saved under build_artifacts/

Method 2 — Using pixi (modern approach)

conda install -c conda-forge pixi
pixi run build-linux

Testing via a local channel


# From the repo root
conda create -y -n test-grass -c file://$PWD/build_artifacts -c conda-forge python=3.12 grass=8.4.1

# Basic runtime checks
conda run -n test-grass grass --version
conda run -n test-grass grass --tmp-project EPSG:4326 --exec g.version -rge

# Download sample data for validation
wget https://grass.osgeo.org/sampledata/north_carolina/nc_basic_spm_grass7.zip
unzip nc_basic_spm_grass7.zip
cd nc_basic_spm_grass7

# Run basic GRASS GIS commands
conda run -n test-grass grass --text user1 --exec g.region -p
conda run -n test-grass grass --text user1 --exec r.info elevation
conda run -n test-grass grass --text user1 --exec g.list type=raster

Next Steps

The maintainers will review the PR and may suggest refinements. Once approved and merged, a new GRASS feedstock repository will be automatically created. After acceptance, @HuidaeCho and I’ll help maintain the feedstock — handling version updates, dependency management, and future build fixes.

For platform coverage:

  • macOS support will be added for the current 8.4.1 release using the same testing and CI validation methods described above.

  • Windows support is planned for the next version (8.5), once environment and dependency adjustments are finalized.

These platforms can be tested using the same approach as above, but note:

  • Linux builds use Docker containers (via build-locally.py).

  • Windows and macOS builds run natively on the host machine since Docker-based builds are not supported there. The host environment must match the target platform (e.g., Windows host for Windows builds, macOS host for macOS builds).

dudaka avatar Oct 19 '25 06:10 dudaka

The CI was passing on this 2 weeks ago already without a human looking at this, so perhaps polishing the PR and then contacting team is the right thing to do. The contacting etiquette is not completely clear to me. Anyway, I left some comments in the PR.

https://conda-forge.org/docs/maintainer/adding_pkgs/#feedback-and-revision https://conda-forge.org/docs/maintainer/maintainer_faq/#how-can-i-contact-conda-forgecore

wenzeslaus avatar Oct 29 '25 13:10 wenzeslaus

@wenzeslaus I started a discussion on Zulip on October 21 but haven’t received any response yet.

#general > Review request: GRASS GIS recipe PR (#31234)

There’s also another thread for other package, #general > ✔ PR that has been wating a review for a month, but that one has already been resolved.

dudaka avatar Oct 29 '25 16:10 dudaka

See https://conda-forge.org/docs/maintainer/infrastructure/#conda-forge-admin-please-ping-team, post the following for the bot:

@conda-forge-admin, please ping @conda-forge/help-python-c

petrasovaa avatar Oct 30 '25 15:10 petrasovaa

@petrasovaa Thanks for the link! That page is used after the recipe PR is merged to manage the feedstock. For now, we just wait for the staged-recipes PR to be approved. Once it's merged, we can use those tools.

I have posted another message to follow up on the zulip chat

dudaka avatar Oct 30 '25 15:10 dudaka

You need to tag the appropriate team in the PR, read the repo Readme, FAQ 12

@petrasovaa Thanks for the link! That page is used after the recipe PR is merged to manage the feedstock. For now, we just wait for the staged-recipes PR to be approved. Once it's merged, we can use those tools.

I have posted another message to follow up on the zulip chat

petrasovaa avatar Oct 30 '25 16:10 petrasovaa

@petrasovaa Thank you!

I posted the comment and the post responded https://github.com/conda-forge/staged-recipes/pull/31234#issuecomment-3469323983

dudaka avatar Oct 30 '25 17:10 dudaka