Support Python 3.11

[EwoutH]

Currently Python 3.11 isn't fully supported in Fiona yet, including testing in CI and publishing wheels to PyPI.

Python 3.11 is expected to be released as stable in October 2022, with many new features including:

• PEP 657 -- Include Fine-Grained Error Locations in Tracebacks
• PEP 654 -- Exception Groups and except*
• PEP 673 -- Self Type
• PEP 646-- Variadic Generics
• PEP 680-- tomllib: Support for Parsing TOML in the Standard Library
• PEP 675-- Arbitrary Literal String Type
• PEP 655-- Marking individual TypedDict items as required or potentially-missing
• bpo-46752-- Introduce task groups to asyncio
• The Faster Cpython Project is already yielding some exciting results: this version of CPython 3.11 is ~ 19% faster on the geometric mean of the PyPerformance benchmarks, compared to 3.10.0.

Fiona is an import package in the geospatial stack, so early support will help speed up Python 3.11 adoption. The last alpha release in the 3.11 series, Python 3.11.0a7, has been released earlier this week, and early May the first beta release will be published, after which no new features will be added.

I think it would be an amazing feat to have full Python 3.11 support in Fiona by the time Python 3.11.0 beta 1 gets released, (expected Friday May 6th, 2022). This includes testing in CI and publishing wheels to PyPI.

[rbuffat]

Fiona will most likely not directly implement Features of Python 3.11 to maintain backward compatibility. From history, the highest probability to cause compatibility issues with new Python versions is Cython. Adding Python 3.11 to the CI makes sense towards the final release. Typically, new packages are uploaded to PyPI with each new release of Fiona. Meanwhile, it is possible to build Fiona from source.

[dennytron]

Hi there -- I'm testing an install of Fiona with Python 3.11 beta 1, and I ran into an issue.

Software Being Used

• Linux (in Docker for MacOS)
• Python 3.11 Beta 1
• Fiona 1.8.21

Issue Fiona won't pip install under Python 3.11. I receive an error, which isn't present with 3.10:

To Reproduce Here is the Dockerfile I'm using:

FROM osgeo/gdal:ubuntu-full-3.2.2 

RUN apt-get update -y 
RUN apt-get install software-properties-common -y 
RUN apt-get install gcc -y

RUN add-apt-repository ppa:deadsnakes/ppa -y 
RUN apt-get update -y
RUN apt-get install python3.11 -y 
RUN apt-get install python3.11-dev -y 

RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py 
RUN apt-get install python3.11-distutils -y 
RUN python3.11 get-pip.py 
RUN python3.11 -m pip install --upgrade pip 

RUN GDAL_CONFIG=/usr/bin/gdal-config python3.11 -m pip install fiona==1.8.21 

Here is the docker build command I'm using: Desktop % docker build . -t test-py311

It could be due to CPython header files moving around in 3.11: https://github.com/cython/cython/issues/4461

Thanks!

[mwtoews]

The issue in the previous comment is that source distribution for Fiona 1.8.21 includes the cythonized *.c files, done by a previous version of Cython that was unaware of future changes. These C files should be excluded from the source distribution (specified in MANIFEST.in), so that they can be cythonized by the version that is present.

[mwtoews]

@dennytron some progress has been done. Some changes to the Dockerfile would add:

RUN apt-get install g++ -y

and then another attempt with:

RUN GDAL_CONFIG=/usr/bin/gdal-config python3.11 -m pip install fiona==1.9a1

(or use pip's --pre option). However, it stumbles on the same error due to use of an older Cython version. From what I've read from others, "Cython 0.29.29 is out and expected to solve some py311 issues."

@sgillies is it necessary to pin Cython here? Or should it be unpinned? https://github.com/Toblerity/Fiona/blob/b3be31fbe699d803c69bbb1e1166220fdb966da5/pyproject.toml#L3

[mwtoews]

Good news, this issue is resolved by the fiona==1.9a2 pre-release, which can be built for Python 3.11

[EwoutH]

#1145 helps unblock building for Python 3.11.

Good news, this issue is resolved by the fiona==1.9a2 pre-release, which can be built for Python 3.11

Awesome! Could we get wheel building for Python 3.11 going on the 1.9 branch? Even just wheels for pre-releases would help the ecosystem.

[EwoutH]

The large_image_wheels project now has Python 3.11 wheels for manylinux available, including the latest GDAL (3.5.2).

I would love to see #1145 reviewed, and try how far I can get in building Python 3.11 wheels.

[EwoutH]

@sgillies Thanks for merging #1145!

The next step is #1147, in which a Python 3.11 job is added to the Linux CI. And it passed!

I also have a questions about the wheel building. The docs state that's done with AppVeyor, is that still the case? The configuration files indicates that only wheels up to Python 3.8 are build?

So what would be needed to test wheel building for Python 3.11?

[sgillies]

Fiona wheels are built by https://github.com/sgillies/fiona-wheels. I'm trying 3.11 at its sister project https://github.com/rasterio/rasterio-wheels/pull/93 and will copy changes over to fiona-wheels when I succeed.

[EwoutH]

I saw https://github.com/rasterio/rasterio-wheels/pull/93 was merged, awesome! How is it going on the Fiona wheels?

[EwoutH]

Is see that there are Python 3.11 wheels on PyPI for both Fiona 1.8.22 and 1.9a3, which is awesome! Thanks for the effort!