Official Windows binary wheels?

[idanmiara]

@rouault, @snowman2 and all, As gdal cannot be installed easily on windows using normal pypi without the use of unofficial binary wheels, Would you please estimate the effort and/or budget that would be required to build and release official windows binary wheels in a similar manner to pyproj?

[snowman2]

pyproj windows wheels come from: https://www.lfd.uci.edu/~gohlke/pythonlibs/

pygeos might be a good reference for how to do it with cibuildwheel: https://github.com/pygeos/pygeos/pull/365

[rouault]

https://github.com/OSGeo/gdal/issues/3060 discussed that (more specifically for Linux), but my thoughts in https://github.com/OSGeo/gdal/issues/3060#issuecomment-719794405 also apply

[idanmiara]

Hi @rouault @snowman2 @rcoup and all,

Given that Christoph Gohlke's Windows Wheels site is no longer maintained (last GDAL version published was 3.4.3) and following a related discussion on the mailing list - I was wondering if there is a new source for new GDAL windows wheels ?

https://lists.osgeo.org/pipermail/gdal-dev/2021-February/053484.html https://www.reddit.com/r/Python/comments/vcaibq/christoph_gohlkes_windows_wheels_site_is_shutting/ https://stackoverflow.com/questions/72581592/what-to-do-when-gohlkes-python-wheel-service-shuts-down https://www.lfd.uci.edu/~gohlke/pythonlibs/ http://eturnbull.ca/pythonlibs/

[rcoup]

I was wondering if there is a new source for new GDAL windows wheels ?

Not that I know of. Rasterio & fiona have Windows binary wheels c/- the hard work of @sgillies

If you wanted to tackle it, my suggestion would be:

1. use vcpkg to build GDAL's dependencies (GDAL is in vcpkg, but only a single version). I'd suggest using static linking. If there are more dependencies you want (eg OpenFileGDB), add them into vcpkg.
2. build the GDAL library & use static linking if possible
3. then create GDAL wheels using cibuildwheel.
4. if you didn't manage to get it working with static linking, then use some sort of custom Windows equivalent to auditwheel/delocate to embed the other DLLs (see https://github.com/pypa/cibuildwheel/issues/191)

[idanmiara]

I see that rasterio (>=1.3.0) and fiona (>=1.8.22) publishe full official Windows wheels and actually including a full gdal and proj installations + data inside those wheels (not sure where is the source code for that).

I'm guessing that this move also had to do with the fact Christoph Gohlke is not making his gdal , rasterio, fiona wheels anymore - the timing seems right: (edit: Christoph posted a link to his new gdal windows wheels) https://github.com/rasterio/rasterio/pull/2709

Maybe we can take the same approach with gdal binary wheels?

Another idea for combining efforts: (@sgillies @rouault @snowman2 @rcoup @hobu - This is a big change and I'm not sure how complicated to implement, but I believe all 3 projects will benefit from this joint effort):

Split the rasterio + fiona wheels (similar approach we did with gdal and gdal_utils) and combine with the gdal swig wrappers so we'll have 3 packages:

1. gdal (platform dependent) - includes gdal and proj binaries+data, rasterio+fiona cython wrappers (as of the new rasterio/fiona wheels, but without the pure python rasterio/fiona code) + gdal swig wrappers + gdal binary utils (i.e. gdalinfo).
2. rasterio (platform independent) - rasterio pure python code - depends on gdal
3. fiona (platform independent) - fiona pure python code - depends on gdal
4. gdal_utils (platform independent) - no change - depends on gdal

From a rasterio/fiona user perspective - pip install rasterio|fiona would install rasterio|fiona and gdal so it would be transparent. From a gdal user perspective - pip install gdal would install gdal (with or without gdal_utils ?)

At this point I guess we also should discuss if we continue to embed the gdal_utils package inside the gdal wheel. If yes - pip install rasterio|fiona would actually install rasterio/fiona+gdal+gdal_utils[embedded in gdal wheel]. If no - a gdal user should be running pip install gdal_utils which would also install gdal (otherwise by running pip install gdal they won't get gdal_utils)

Advantages:

1. Using a single gdal+proj installtaion/data (pip install gdal rasterio fiona would otherwise install 3 different versions of gdal/proj)
2. Automating gdal for windows builds
3. Much smaller wheels for rasterio and fiona (a few KB instead of ~22MB)

Disadvantages/Issues:

1. Won't be possible to install rasterio|fiona that depend on different versions of gdal
2. Licensing difference between the cython bindings and swig bindings
3. Compatibility issues between the cython bindings inside gdal wheel of rasterio|fiona version X and rasterio|fiona version Y (also might be an advantage to decouple between the bindings and the python code ?)

Also should be discussed how this effects the MacOS and Linux users (I'm only familiar with the wheels structure on Windows)

[cgohlke]

I was wondering if there is a new source for new GDAL windows wheels ?

https://github.com/cgohlke/gdal.whl/releases

[SamRWest]

I was wondering if there is a new source for new GDAL windows wheels ?

https://github.com/cgohlke/gdal.whl/releases

This is a really useful stopgap, and I applaud @cgohlke for the time and effort involved, but I hope that it isn't being considered a reasonable substitute for this project publishing wheels to PyPi?

[jratike80]

The GDAL ecosystem is fortunately larger than just the bunch of core developers. Likewise the Windows binaries are published by others (OSGeo4W, gisinternals.com, MapServer etc.). I think that those, and other, three party projects are reasonable substitutes because if the core developers would do more packaging they would do less something else.

[idanmiara]

I'm thinking that if someone would make a similar pipeline to the ones used by rasterio/fiona it might be accepted as the official GDAL wheels on pypi. https://github.com/rasterio/rasterio-wheels https://github.com/sgillies/fiona-wheels

raise your hand if you are that someone!

On Tue, 6 Jun 2023 at 09:54, Jukka Rahkonen @.***> wrote:

The GDAL ecosystem is fortunately larger than just the bunch of core developers. Likewise the Windows binaries are published by others (OSGeo4W, gisinternals.com, MapServer etc.). I think that those, and other, three party projects are reasonable substitutes because if the core developers would do more packaging they would do less something else.

— Reply to this email directly, view it on GitHub https://github.com/OSGeo/gdal/issues/4352#issuecomment-1578026268, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGJBBLNWLMU6AMEUTLNYUI3XJ3H37ANCNFSM5C4WICEA . You are receiving this because you authored the thread.Message ID: @.***>

[apwebber]

The GDAL ecosystem is fortunately larger than just the bunch of core developers. Likewise the Windows binaries are published by others (OSGeo4W, gisinternals.com, MapServer etc.). I think that those, and other, three party projects are reasonable substitutes because if the core developers would do more packaging they would do less something else.

If this new repo is going to be relied upon it needs to be linked to. Currently only the conda way of installing is mentioned. It is very much by chance that people find these wheels.

[connorjak]

https://www.lfd.uci.edu/~gohlke/pythonlibs/ is now down; it seems https://github.com/cgohlke/geospatial-wheels/releases is the replacement