geopandas / fiona not imported properly in Python 3.7 env

[David-Loibl]

I have installed OGGM following the guide with Python 3.7 (conda create --name oggm_env python=3.7). When I run the steps from the Getting Started Tutorial I get the following error when trying to create the gdirs:

[IN] gdirs = workflow.init_glacier_regions(rgi_ids, from_prepro_level=2, prepro_border=80)

[OUT]
2019-10-10 15:32:46: oggm.workflow: init_glacier_regions from prepro level 2 on 2 glaciers.
2019-10-10 15:32:46: oggm.workflow: Execute entity task gdir_from_prepro on 2 glaciers
100% of 230.0 KiB |####################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
100% of 530.0 KiB |####################################################################################################| Elapsed Time: 0:00:00 Time:  0:00:00
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/oggm/workflow.py", line 92, in __call__
    return call_func(gdir, **self.out_kwargs)
  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/oggm/workflow.py", line 217, in gdir_from_prepro
    return oggm.GlacierDirectory(entity, from_tar=from_tar)
  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/oggm/utils/_workflow.py", line 1530, in __init__
    rgi_entity = self._read_shapefile_from_path(_shp)
  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/oggm/utils/_workflow.py", line 1899, in _read_shapefile_from_path
    return gpd.read_file(fp)
NameError: name 'gpd' is not defined
""" 
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/oggm/workflow.py", line 319, in init_glacier_regions
    check_demo_glacier=use_demo_glaciers)
  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/oggm/workflow.py", line 147, in execute_entity_task
    out = mppool.map(pc, gdirs, chunksize=1)
  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
NameError: name 'gpd' is not defined `

Looking into utils/_workflow.py I realized that geopandas was loaded only optionally by a try statement. After changing this to import geopandas as gpd the following error pops up:

[IN] gdirs = workflow.init_glacier_regions(rgi_ids, from_prepro_level=2, prepro_border=80)

[OUT]
Traceback (most recent call last):

  File "<ipython-input-1-9f5ddc29827f>", line 1, in <module>
    from oggm import cfg, utils

  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/oggm/__init__.py", line 26, in <module>
    from oggm.utils import entity_task, global_task, GlacierDirectory

  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/oggm/utils/__init__.py", line 4, in <module>
    from oggm.utils._workflow import *

  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/oggm/utils/_workflow.py", line 36, in <module>
    import geopandas as gpd

  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/geopandas/__init__.py", line 5, in <module>
    from geopandas.io.file import read_file  # noqa

  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/geopandas/io/file.py", line 6, in <module>
    import fiona

  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/fiona/__init__.py", line 83, in <module>
    from fiona.collection import BytesCollection, Collection

  File "/home/loibldav/anaconda3/envs/oggm_env/lib/python3.7/site-packages/fiona/collection.py", line 9, in <module>
    from fiona.ogrext import Iterator, ItemsIterator, KeysIterator

ImportError: libpoppler.so.76: cannot open shared object file: No such file or directory

Manually checking geopandas and fiona with conda did not help, both are installed and up-to-date according to conda.

I have tested this on two computers, same errors on both machines. Notably, everything works fine when choosing Python 3.6 when creating the conda environment.

[TimoRoth]

This is partially conda being terrible at dependency resolution and combining incompatible binaries by default, and partially oggm silently ignoring failing imports of a few optional packages.

Try using one of the pre-tested environments: https://github.com/OGGM/OGGM-dependency-list/blob/master/linux-64/oggmdev-latest_py37.yml

[David-Loibl]

Thanks for the swift reply and the recommendation. Maybe this hint should be added to the setup instructions, where it sounds like setting up a Python 3.7 conda environment was tested to be fully working? For me everything works fine with a Python 3.6 env at the moment. Personally, I would not consider a package that leads to failure of a key feature 'optional', though.

[TimoRoth]

It was tested to be fully working, but anaconda/conda-forge changes stuff almost daily and it's impossible to keep up. Old packages are also frequently deleted, so giving a fixed "known working" environment would quickly lead to a non working install as well.