TypeError on import with GDAL version 3.1+

[tomplex]

Expected behavior and actual behavior.

import fiona succeeds without error when built against the most recent version of GDAL. However with GDAL 3.1.2 and 3.2.0, there is a TypeError on import.

Steps to reproduce the problem.

FROM osgeo/gdal

RUN apt update && \
    apt install -y \
    python3-pip

RUN pip3 install --no-binary fiona fiona

ENTRYPOINT ["python3", "-c", "import fiona"]

docker build -t test-fiona .
docker run test-fiona

This results in a TypeError:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/fiona/__init__.py", line 86, in <module>
    from fiona.collection import BytesCollection, Collection
  File "/usr/local/lib/python3.8/dist-packages/fiona/collection.py", line 11, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator
  File "fiona/ogrext.pyx", line 1, in init fiona.ogrext
TypeError: C function fiona._shim.get_field_as_datetime has wrong signature (expected __pyx_ctuple_e69d__int__and_int__and_int__and_int__and_int__and_int__and_float__and_int__etc (void *, int), got __pyx_ctuple_1cb47__int__and_int__and_int__and_int__and_int__and_int__and_float__and_int__etc (void *, int))

Operating system & GDAL version

I first observed this issue on Debian Stretch using GDAL 3.1.2, but I can't share that image as it's work-related. Fortunately I was able to replicate using the latest GDAL images.

# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal

# gdal-config --version
3.2.0

# ogr2ogr --version
GDAL 3.2.0dev-603eb8ac0327b25c8b85ad8ef8958290405482ef, released 2020/09/03

[jorisvandenbossche]

We are seeing the same error on the conda-forge builds: https://github.com/conda-forge/fiona-feedstock/pull/166 And indeed, the GDAL version might be it, on conda-forge it is building with GDAL 3.1.2. Locally it was working for me, but it seems I was using GDAL 3.0.4. After updating to 3.1.2, it also fails locally.

[rbuffat]

I would be surprised if this is related to a GDAL version, as Fiona uses the same code for all GDAL 3.x variants. On travis and appveyor, fiona builds against gdal 3.0.4, gdal 3.1.2 and gdal master without showing this issue.

The culprit is this function: https://github.com/Toblerity/Fiona/blob/maint-1.8/fiona/_shim3.pyx#L180-L192 returning tuples. From the error message, it seems as there are two tuple definitions created by cython, thus I assume it is something related to cython.

 __pyx_ctuple_e69d__int__and_int__and_int__and_int__and_int__and_int__and_float__and_int__etc (void *, int), got 
__pyx_ctuple_1cb47__int__and_int__and_int__and_int__and_int__and_int__and_float__and_int__etc (void *, int)

Would it be possible to share the generated _shim.c? Also, it would be good to know which cython versions cause this issue. (Or if every cython version > 0.29.17 causes issues, then this would be also useful to know)

[jorisvandenbossche]

With cython 0.29.21, the generated shim.c files for GDAL 3.0.4 vs 3.1.2: _shim.zip (together in a zip file, can't upload c files as is) The only difference between the two files is the hash in the "_pyx_ctuple_7aff8..." name in the signature in the function.

[jorisvandenbossche]

If I then downgrade cython to 0.29.14, and try to build with GDAL 3.1.2, I get the following build error:

gcc -pthread -B /home/joris/miniconda3/envs/geo-dev-fiona/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/joris/miniconda3/envs/geo-dev-fiona/include -I/home/joris/miniconda3/envs/geo-dev-fiona/include/python3.8 -c fiona/_env.c -o build/temp.linux-x86_64-3.8/fiona/_env.o
fiona/_env.c:1503:1: warning: parameter names (without types) in function declaration
 static PyObject* __pyx_convert__to_py___pyx_ctuple_int__and_int__and_int(__pyx_ctuple_int__and_int__and_int);
 ^~~~~~
fiona/_env.c:1543:44: error: expected declaration specifiers or ‘...’ before ‘*’ token
 static __pyx_ctuple_int__and_int__and_int (*__pyx_f_5fiona_5_shim_get_proj_version)(void); /*proto*/
                                            ^
fiona/_env.c: In function ‘__pyx_pf_5fiona_4_env_8get_proj_version_tuple’:
fiona/_env.c:2928:73: warning: implicit declaration of function ‘__pyx_f_5fiona_5_shim_get_proj_version’; did you mean ‘__pyx_f_5fiona_5_shim_set_proj_search_path’? [-Wimplicit-function-declaration]
     __pyx_t_1 = __pyx_convert__to_py___pyx_ctuple_int__and_int__and_int(__pyx_f_5fiona_5_shim_get_proj_version()); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 125, __pyx_L1_error)
                                                                         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                                                         __pyx_f_5fiona_5_shim_set_proj_search_path
fiona/_env.c: In function ‘__Pyx_modinit_function_import_code’:
fiona/_env.c:11703:77: error: ‘__pyx_f_5fiona_5_shim_get_proj_version’ undeclared (first use in this function); did you mean ‘__pyx_f_5fiona_5_shim_set_proj_search_path’?
   if (__Pyx_ImportFunction(__pyx_t_2, "get_proj_version", (void (**)(void))&__pyx_f_5fiona_5_shim_get_proj_version, "__pyx_ctuple_int__and_int__and_int (void)") < 0) __PYX_ERR(0, 1, __pyx_L1_error)
                                                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                                                             __pyx_f_5fiona_5_shim_set_proj_search_path
fiona/_env.c:11703:77: note: each undeclared identifier is reported only once for each function it appears in
fiona/_env.c: At top level:
fiona/_env.c:15837:74: error: unknown type name ‘__pyx_ctuple_int__and_int__and_int’
 static PyObject* __pyx_convert__to_py___pyx_ctuple_int__and_int__and_int(__pyx_ctuple_int__and_int__and_int value) {
                                                                          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
fiona/_env.c:1503:18: warning: ‘__pyx_convert__to_py___pyx_ctuple_int__and_int__and_int’ used but never defined
 static PyObject* __pyx_convert__to_py___pyx_ctuple_int__and_int__and_int(__pyx_ctuple_int__and_int__and_int);
                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
error: command 'gcc' failed with exit status 1

Is this expected the ctuple return only works with more recent cython versions?

[rbuffat]

@jorisvandenbossche I think this was fixed with cython 0.29.16: https://github.com/cython/cython/pull/3271

[jorisvandenbossche]

Then with cython 0.29.17, it builds but fails importing. (btw, I get a different error on import locally, "AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)", so not the explicit one about the ctuple signature mismatch)

[rbuffat]

@jorisvandenbossche if you delete all *.c files and recompile everything, does it still produce the same issue?

[jorisvandenbossche]

Ah, I was using python setup.py clean, but that doesn't seem to clean-up c file (they still seem to be recompiled though).

I think this was fixed with cython 0.29.16: cython/cython#3271

Yes, so with 0.29.16, same thing (builds fine, but import fails), and with 0.29.15 the build fails as with 0.29.14.

And then with ensuring all .c files were cleaned-up before rebuilding (git clean -xdf), and using cython 0.29.17, it's now working .. (still with GDAL 3.1.2). But I don't understand why it would then fail on eg conda-forge (where I assume it is also doing a clean build)

[jorisvandenbossche]

And also with the latest 0.29.21, it is working fine now locally when ensuring a fully clean build ..

[rbuffat]

@sgillies @jorisvandenbossche @tomplex The tar.gz from https://pypi.org/project/Fiona/#files contains precompiled c files. This allows the source distribution not to depend on cython. I suspect each _shim generates a different tuple typedef. Thus the precompiled ogrext.c works only with one of the precompiled _shim.c.

As a workaround, if cython is available, all c files should be deleted before installing Fiona. Fiona should be patched so that either cython generates always the same typedef (not sure if possible or how to) or that get_field_as_datetime does not return tuples.

[sgillies]

@rbuffat thanks for the analysis! I merged #942, will test it here, and will release a 1.8.15 later this morning (US Mountain time).

[tomplex]

Thanks for the quick analysis and fix everyone!

[jorisvandenbossche]

WIth fiona 1.8.15, the issue is fixed now in the conda-forge builds, only the Windows py3.8 build is still failing: https://github.com/conda-forge/fiona-feedstock/pull/167

The failure is with the general error message of partially initialized module, so doesn't give much pointers (at least to me):

running run_test.py
Traceback (most recent call last):
  File "D:\bld\fiona_1599200296823\test_tmp\run_test.py", line 6, in <module>
    import fiona
  File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\__init__.py", line 84, in <module>
    import fiona._loading
  File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\_loading.py", line 44, in <module>
    import fiona.ogrext
  File "fiona/ogrext.pyx", line 29, in init fiona.ogrext
  File "D:\bld\fiona_1599200296823\_test_env\lib\site-packages\fiona\env.py", line 14, in <module>
    with fiona._loading.add_gdal_dll_directories():
AttributeError: partially initialized module 'fiona' has no attribute '_loading' (most likely due to a circular import)

[sgillies]

We've fixed the partially initialized module issue: https://github.com/Toblerity/Fiona/blob/master/CHANGES.txt#L57.