Fiona 1.8.21 breaks GDAL/PROJ environment in Docker/Virtualenv

[remi-braun]

Hello,

Installing Fiona 1.8.21 breaks my GDAL/Proj environment in my docker image, with Python 3.7 running on Debian:buster-slim. However, rolling back to 1.8.20 with the exact same Dockerfile makes it work again.

I have several different errors like:

• fiona._err.CPLE_AppDefinedError: PROJ: proj_create: no database context specified
• fiona._err.CPLE_AppDefinedError: Cannot import GDLIB:32701 due to ALLOW_FILE_ACCESS=NO
• PROJCS["WGS 84 / UTM zone 40N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",57],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32640"]] != epsg:4326

Did you changed the gdal_data directory of Fiona ?

Here are the way I export my GDAL environment variables:

ENV CPLUS_INCLUDE_PATH="/usr/include/gdal"
ENV C_INCLUDE_PATH="/usr/include/gdal"
ENV PROJ_LIB="/usr/local/lib/python3.7/dist-packages/pyproj/proj_dir/share/proj"
ENV GDAL_DATA="/usr/local/lib/python3.7/dist-packages/fiona/gdal_data/"

[rbuffat]

It is advised not to mix different GDAL installations, e.g. the GDAL included in the Fiona wheels and a system GDAL. The Fiona 1.8.21 wheels include GDAL 3.4.1, the wheels for 1.8.20 include GDAL 3.3.0.

If your setups releys on a specific version or setup of GDAL outside the context of Python, this might have worked only by chance.

[remi-braun]

GDAL isn't included in my requirements (however I'm installing fiona and rasterio) and my Dockerfile looks like that (parts concerning GDAL):

# New image
FROM debian:buster-slim

# Basic libs and install python3.7
RUN apt-get update && apt-get upgrade -y \
&& apt-get install -y procps wget python3.7 python3-pip gdal-bin libgdal-dev \
&& apt-get clean  \
&& rm -rf /var/lib/apt/lists/*

# Pip
RUN python -m pip install --no-cache-dir --upgrade pip \
&& pip install --no-cache-dir --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL==`gdal-config --version` \
&& pip install --no-cache-dir --ignore-installed PyYAML \
&& pip install --no-cache-dir --upgrade -r requirements.txt

# Setting some ENV variables
ENV CPLUS_INCLUDE_PATH="/usr/include/gdal"
ENV C_INCLUDE_PATH="/usr/include/gdal"
ENV PROJ_LIB="/usr/local/lib/python3.7/dist-packages/pyproj/proj_dir/share/proj"
ENV GDAL_DATA="/usr/local/lib/python3.7/dist-packages/fiona/gdal_data/"

You seem to say that I shouldn't do pip install --no-cache-dir --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL=='gdal-config --version', am I right ?

[sgillies]

@remi-braun is your intention to have rasterio and fiona use the same GDAL apt package? In that case, eliminate pip install --no-cache-dir --global-option=build_ext --global-option="-I/usr/include/gdal" GDAL=='gdal-config --version' and replace with pip install --no-binary fiona,rasterio fiona rasterio. Fiona and rasterio don't depend on the GDAL Python package. Then you can unset PROJ_LIB and GDAL_DATA.

The GDAL and PROJ data files included in the fiona and rasterio wheels are intended for use with that installed package only. They can't reliably be shared with other applications on your system.

[remi-braun]

Thanks a lot for the tip, I would never have found that 😅

[remi-braun]

Just to say, I have exactly the same issue (same errors) with tox venv and I am not able to fix it 😓 (and it was working before the 1.8.21)

I am using the Ubuntu, running tox with this tox.ini configuration file:

[tox]
envlist = py37, py38, py39, linters
skipdist = true

[gh-actions]
python =
    3.9: py39
    3.8: py38
    3.7: py37

[testenv]
; Install requirements
allowlist_externals =
    python
    sudo
    apt-get
commands_pre =
    sudo apt-get update
    sudo apt-get upgrade -y
    sudo apt-get install -y python-numpy gdal-bin libgdal-dev
    python -m pip install --no-binary fiona,rasterio -r{toxinidir}{/}requirements.txt

; Run Pytest
commands = pytest -v --durations=0 --cov-report term --cov-report xml:cov.xml --cov=sertit CI{/}SCRIPTS

; Environment variables
setenv =
    TMPDIR={envtmpdir}
    HOME={envtmpdir}
    CPLUS_INCLUDE_PATH=/usr/include/gdal
    C_INCLUDE_PATH=/usr/include/gdal

I tried to mimic my Dockerfile but it still won't work...

[sgillies]

@remi-braun I overlooked one thing in your initial report:

fiona._err.CPLE_AppDefinedError: Cannot import GDLIB:32701 due to ALLOW_FILE_ACCESS=NO

According to https://gdal.org/doxygen/classOGRSpatialReference.html#a40d5f1ce7fbd23a6c35ab9d2473cd6e2, the default for this configuration option is "YES". Do you understand why it is "NO" in your case?

@rouault I see this is related to a change in https://github.com/OSGeo/gdal/pull/4193. Does use of a particular driver(s) affect this config option?

To be clear, this issue does not surface in Fiona's tests. However, we are exercising only a subset of drivers and access modes.

[remi-braun]

Nope I don't understand why it is NO and I tried to change it but I haven't found how.

However it seems linked to the GDAL environment because your workaround for the Dockerfile has solved it all.

Why it doesnt work in tox is a mistery for me 😓

[rouault]

@remi-braun what "GDLIB:32701" is supposed to mean ? Is GDLIB a PROJ.4 style init file and 32701 a code in it ? And how does it reach OGRSpatialReference::SetFromUserInput() ? From the error message and OSGeo/gdal#4193, as @sgillies pointed out, it seems that it would come from a CRS definition read by a GDAL driver ?

[remi-braun]

@remi-braun what "GDLIB:32701" is supposed to mean ? Is GDLIB a PROJ.4 style init file and 32701 a code in it ? And how does it reach OGRSpatialReference::SetFromUserInput() ? From the error message and OSGeo/gdal#4193, as @sgillies pointed out, it seems that it would come from a CRS definition read by a GDAL driver ?

Yes it seems so, those errors do seem to always come from crs reading. It is very strange as it doesn't happen every time, some tests are just passing. And it happens only with vectors, not rasters (so rasterio seems to work fine)

Another similar error can be found here, with a geometry with no CRS (and GML driver) ➤ this is the only public test I have with this error

[remi-braun]

It seems most of the errors comes from GML files with unknown CRS (that should be known)

Here are some exmples: ROI_SPOT7_MS_201602150257025_SEN_1671661101_MSK.zip ROI_SPOT6_PMS_201812180938485_ORT_3726409101_MSK.zip ROI_PNEO3_202201182152117_PMS_ORT_PWOI_000007338_1_1_F_1.zip

As you see, it is always Airbus Masks (Sentinel-2 GMLs are working just fine)

[Hugovdberg]

I have the same issue with reading GML files, but not with a ESRI Shapefile with a prj file. Is this an issue that GDAL tries to read files not related directly to the sourcefile? The 28992 is the EPSG code for the Dutch Rijksdriehoek coordinate system.

---------------------------------------------------------------------------
CPLE_AppDefinedError                      Traceback (most recent call last)
c:\Temp\perceel_kruisland.ipynb Cell 4' in <cell line: [1](vscode-notebook-cell:/c%3A/Temp/perceel_kruisland.ipynb#ch0000001?line=0)>()
----> 1 gpd.read_file("dkk_perceel.gml")

File c:\Temp\conda-envs\imod\lib\site-packages\geopandas\io\file.py:208, in _read_file(filename, bbox, mask, rows, **kwargs)
    [200] with fiona_env():
    [201]     with reader(path_or_bytes, **kwargs) as features:
    [202] 
    [203]         # In a future Fiona release the crs attribute of features will
    [204]         # no longer be a dict, but will behave like a dict. So this should
    [205]         # be forwards compatible
    [206]         crs = (
    [207]             features.crs["init"]
--> [208]             if features.crs and "init" in features.crs
    [209]             else features.crs_wkt
    [210]         )
    [212]         # handle loading the bounding box
    [213]         if bbox is not None:

File c:\Temp\conda-envs\imod\lib\site-packages\fiona\collection.py:215, in Collection.crs(self)
    [213] """Returns a Proj4 string."""
    [214] if self._crs is None and self.session:
--> [215]     self._crs = self.session.get_crs()
    [216] return self._crs

File fiona\ogrext.pyx:742, in fiona.ogrext.Session.get_crs()

File fiona\_err.pyx:291, in fiona._err.exc_wrap_pointer()

CPLE_AppDefinedError: Cannot import 28992 due to ALLOW_FILE_ACCESS=NO

[sgillies]

This was resolved upstream in GDAL.