cartopy
cartopy copied to clipboard
Published
20 hours ago •
SciTools
SciTools
Key Error using Natural Earth POV files
Description
When trying to use Natural Earth's POV shapefiles, the unzipping step fails because they do not include a .cpg file. I wonder if we just need a try-except around the getinfo call. Having said that, out of the list
https://github.com/SciTools/cartopy/blob/ec891b0f119a5a21f7c404b49e8cc2f84a50ea38/lib/cartopy/io/shapereader.py#L342
we only appear to use .shp and .shx in other parts of the library. So is there a reason we are unpacking all of them?
Code to reproduce
import cartopy
import cartopy.crs as ccrs
import matplotlib.pyplot as plt
mpov = cartopy.feature.NaturalEarthFeature('cultural', 'admin_0_countries_arg', '10m')
fig, ax = plt.subplots(subplot_kw={"projection": ccrs.PlateCarree()})
ax.add_feature(mpov, facecolor="none")
plt.show()
Traceback
On first run, when Cartopy is downloading and unzipping the files I get
[script_path]/natural_earth.py:3: UserWarning: Treat the new Tool classes introduced in v1.5 as experimental for now; the API and rcParam may change in future versions.
import matplotlib.pyplot as plt
[git_clone_path]/cartopy/lib/cartopy/io/__init__.py:241: DownloadWarning: Downloading: https://naturalearth.s3.amazonaws.com/10m_cultural/ne_10m_admin_0_countries_arg.zip
warnings.warn(f'Downloading: {url}', DownloadWarning)
Exception in Tkinter callback
Traceback (most recent call last):
File "[conda_env_path]/cartopy-dev/lib/python3.12/tkinter/__init__.py", line 1962, in __call__
return self.func(*args)
^^^^^^^^^^^^^^^^
File "[conda_env_path]/cartopy-dev/lib/python3.12/tkinter/__init__.py", line 861, in callit
func(*args)
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/backends/_backend_tk.py", line 274, in idle_draw
self.draw()
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/backends/backend_tkagg.py", line 10, in draw
super().draw()
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/backends/backend_agg.py", line 388, in draw
self.figure.draw(self.renderer)
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/artist.py", line 95, in draw_wrapper
result = draw(artist, renderer, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
return draw(artist, renderer)
^^^^^^^^^^^^^^^^^^^^^^
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/figure.py", line 3154, in draw
mimage._draw_list_compositing_images(
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/image.py", line 132, in _draw_list_compositing_images
a.draw(renderer)
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
return draw(artist, renderer)
^^^^^^^^^^^^^^^^^^^^^^
File "[git_clone_path]/cartopy/lib/cartopy/mpl/geoaxes.py", line 524, in draw
return super().draw(renderer=renderer, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
return draw(artist, renderer)
^^^^^^^^^^^^^^^^^^^^^^
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/axes/_base.py", line 3070, in draw
mimage._draw_list_compositing_images(
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/image.py", line 132, in _draw_list_compositing_images
a.draw(renderer)
File "[conda_env_path]/cartopy-dev/lib/python3.12/site-packages/matplotlib/artist.py", line 72, in draw_wrapper
return draw(artist, renderer)
^^^^^^^^^^^^^^^^^^^^^^
File "[git_clone_path]/cartopy/lib/cartopy/mpl/feature_artist.py", line 152, in draw
geoms = self._feature.intersecting_geometries(extent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[git_clone_path]/cartopy/lib/cartopy/feature/__init__.py", line 305, in intersecting_geometries
return super().intersecting_geometries(extent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[git_clone_path]/cartopy/lib/cartopy/feature/__init__.py", line 108, in intersecting_geometries
return (geom for geom in self.geometries() if
^^^^^^^^^^^^^^^^^
File "[git_clone_path]/cartopy/lib/cartopy/feature/__init__.py", line 287, in geometries
path = shapereader.natural_earth(resolution=self.scale,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[git_clone_path]/cartopy/lib/cartopy/io/shapereader.py", line 306, in natural_earth
return ne_downloader.path(format_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[git_clone_path]/cartopy/lib/cartopy/io/__init__.py", line 203, in path
result_path = self.acquire_resource(target_path, format_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[git_clone_path]/cartopy/lib/cartopy/io/shapereader.py", line 364, in acquire_resource
member = zfh.getinfo(member_path.replace('\\', '/'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "[conda_env_path]/cartopy-dev/lib/python3.12/zipfile/__init__.py", line 1516, in getinfo
raise KeyError(
KeyError: "There is no item named 'ne_10m_admin_0_countries_arg.cpg' in the archive"
On second run, the plot works fine.
Full environment definition
Operating system
RHEL7
Cartopy version
0.22 and main
conda list
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
accessible-pygments 0.0.4 pyhd8ed1ab_0 conda-forge
alabaster 0.7.16 pyhd8ed1ab_0 conda-forge
asttokens 2.4.1 pyhd8ed1ab_0 conda-forge
babel 2.14.0 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.12.2 pyha770c72_0 conda-forge
blosc 1.21.5 h0f2a231_0 conda-forge
brotli 1.1.0 hd590300_1 conda-forge
brotli-bin 1.1.0 hd590300_1 conda-forge
brotli-python 1.1.0 py312h30efb56_1 conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
c-ares 1.25.0 hd590300_0 conda-forge
ca-certificates 2023.11.17 hbcca054_0 conda-forge
cartopy 0.22.1.dev89+g5e449397 pypi_0 pypi
certifi 2023.11.17 pyhd8ed1ab_0 conda-forge
cffi 1.16.0 py312hf06ca03_0 conda-forge
cfgv 3.3.1 pyhd8ed1ab_0 conda-forge
cftime 1.6.3 py312hc7c0aa3_0 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
contourpy 1.2.0 py312h8572e83_0 conda-forge
cycler 0.12.1 pyhd8ed1ab_0 conda-forge
cython 3.0.7 py312h30efb56_0 conda-forge
decorator 5.1.1 pyhd8ed1ab_0 conda-forge
distlib 0.3.8 pyhd8ed1ab_0 conda-forge
docutils 0.20.1 py312h7900ff3_3 conda-forge
exceptiongroup 1.2.0 pyhd8ed1ab_0 conda-forge
execnet 2.0.2 pyhd8ed1ab_0 conda-forge
executing 2.0.1 pyhd8ed1ab_0 conda-forge
filelock 3.13.1 pyhd8ed1ab_0 conda-forge
fonttools 4.47.0 py312h98912ed_0 conda-forge
freetype 2.12.1 h267a509_2 conda-forge
geos 3.12.1 h59595ed_0 conda-forge
hdf4 4.2.15 h2a13503_7 conda-forge
hdf5 1.14.3 nompi_h4f84152_100 conda-forge
icu 73.2 h59595ed_0 conda-forge
identify 2.5.33 pyhd8ed1ab_0 conda-forge
idna 3.6 pyhd8ed1ab_0 conda-forge
imagesize 1.4.1 pyhd8ed1ab_0 conda-forge
importlib-metadata 7.0.1 pyha770c72_0 conda-forge
importlib_resources 6.1.1 pyhd8ed1ab_0 conda-forge
iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge
ipython 8.20.0 pyh707e725_0 conda-forge
jedi 0.19.1 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.5 py312h8572e83_1 conda-forge
krb5 1.21.2 h659d440_0 conda-forge
lcms2 2.16 hb7c19ff_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libaec 1.1.2 h59595ed_1 conda-forge
libblas 3.9.0 20_linux64_openblas conda-forge
libbrotlicommon 1.1.0 hd590300_1 conda-forge
libbrotlidec 1.1.0 hd590300_1 conda-forge
libbrotlienc 1.1.0 hd590300_1 conda-forge
libcblas 3.9.0 20_linux64_openblas conda-forge
libcurl 8.5.0 hca28451_0 conda-forge
libdeflate 1.19 hd590300_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 hd590300_2 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h807b86a_3 conda-forge
libgfortran-ng 13.2.0 h69a702a_3 conda-forge
libgfortran5 13.2.0 ha4646dd_3 conda-forge
libgomp 13.2.0 h807b86a_3 conda-forge
libiconv 1.17 hd590300_2 conda-forge
libjpeg-turbo 3.0.0 hd590300_1 conda-forge
liblapack 3.9.0 20_linux64_openblas conda-forge
libnetcdf 4.9.2 nompi_h9612171_113 conda-forge
libnghttp2 1.58.0 h47da74e_1 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libopenblas 0.3.25 pthreads_h413a1c8_0 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libsqlite 3.44.2 h2797004_0 conda-forge
libssh2 1.11.0 h0841786_0 conda-forge
libstdcxx-ng 13.2.0 h7e041cc_3 conda-forge
libtiff 4.6.0 ha9c0a0a_2 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libwebp-base 1.3.2 hd590300_0 conda-forge
libxcb 1.15 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libxml2 2.12.3 h232c23b_0 conda-forge
libxslt 1.1.39 h76b75d6_0 conda-forge
libzip 1.10.1 h2629f0a_3 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
lxml 5.1.0 py312h37b5203_0 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
markupsafe 2.1.3 py312h98912ed_1 conda-forge
matplotlib-base 3.8.2 py312h0e6d468_200 conda-forge/label/testing
matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
ncurses 6.4 h59595ed_2 conda-forge
netcdf4 1.6.5 nompi_py312h26027e0_100 conda-forge
nodeenv 1.8.0 pyhd8ed1ab_0 conda-forge
numpy 1.26.3 py312heda63a1_0 conda-forge
openjpeg 2.5.0 h488ebb8_3 conda-forge
openssl 3.2.0 hd590300_1 conda-forge
owslib 0.29.3 pyhd8ed1ab_0 conda-forge
packaging 23.2 pyhd8ed1ab_0 conda-forge
pandas 2.1.4 py312hfb8ada1_0 conda-forge
parso 0.8.3 pyhd8ed1ab_0 conda-forge
pexpect 4.8.0 pyh1a96a4e_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 10.2.0 py312hf3581a9_0 conda-forge
pip 23.3.2 pyhd8ed1ab_0 conda-forge
platformdirs 4.1.0 pyhd8ed1ab_0 conda-forge
pluggy 1.3.0 pyhd8ed1ab_0 conda-forge
pre-commit 3.6.0 pyha770c72_0 conda-forge
proj 9.3.1 h1d62c97_0 conda-forge
prompt-toolkit 3.0.42 pyha770c72_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pydata-sphinx-theme 0.15.1 pyhd8ed1ab_0 conda-forge
pygments 2.17.2 pyhd8ed1ab_0 conda-forge
pykdtree 1.3.10 py312hc7c0aa3_0 conda-forge
pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge
pyproj 3.6.1 py312h38f1c37_5 conda-forge
pyshp 2.3.1 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
pytest 7.4.4 pyhd8ed1ab_0 conda-forge
pytest-mpl 0.16.1 pyhd8ed1ab_0 conda-forge
pytest-xdist 3.5.0 pyhd8ed1ab_0 conda-forge
python 3.12.1 hab00c5b_1_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-tzdata 2023.4 pyhd8ed1ab_0 conda-forge
python_abi 3.12 4_cp312 conda-forge
pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge
pyyaml 6.0.1 py312h98912ed_1 conda-forge
readline 8.2 h8228510_1 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
ruff 0.1.11 py312h9118e91_0 conda-forge
scipy 1.11.4 py312heda63a1_0 conda-forge
setuptools 69.0.3 pyhd8ed1ab_0 conda-forge
setuptools-scm 8.0.4 pyhd8ed1ab_0 conda-forge
setuptools_scm 8.0.4 hd8ed1ab_0 conda-forge
shapely 2.0.2 py312h9e6bd2c_1 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 h9fff704_0 conda-forge
snowballstemmer 2.2.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.5 pyhd8ed1ab_1 conda-forge
sphinx 7.2.6 pyhd8ed1ab_0 conda-forge
sphinx-gallery 0.15.0 pyhd8ed1ab_0 conda-forge
sphinxcontrib-applehelp 1.0.7 pyhd8ed1ab_0 conda-forge
sphinxcontrib-devhelp 1.0.5 pyhd8ed1ab_0 conda-forge
sphinxcontrib-htmlhelp 2.0.4 pyhd8ed1ab_0 conda-forge
sphinxcontrib-jsmath 1.0.1 pyhd8ed1ab_0 conda-forge
sphinxcontrib-qthelp 1.0.6 pyhd8ed1ab_0 conda-forge
sphinxcontrib-serializinghtml 1.1.9 pyhd8ed1ab_0 conda-forge
sqlite 3.44.2 h2c6b66d_0 conda-forge
stack_data 0.6.2 pyhd8ed1ab_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
traitlets 5.14.1 pyhd8ed1ab_0 conda-forge
typing-extensions 4.9.0 hd8ed1ab_0 conda-forge
typing_extensions 4.9.0 pyha770c72_0 conda-forge
tzdata 2023d h0c530f3_0 conda-forge
ukkonen 1.0.1 py312h8572e83_4 conda-forge
urllib3 2.1.0 pyhd8ed1ab_0 conda-forge
virtualenv 20.25.0 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge
wheel 0.42.0 pyhd8ed1ab_0 conda-forge
xarray 2023.12.0 pyhd8ed1ab_0 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zipp 3.17.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.5 hfc55251_0 conda-forge
pip list
OK, I see the .prj files were deliberately added to help geopandas: #1744, #1745, and .cpg came in with that.
Adding a try/except seems reasonable. I'm a bit confused as to why we don't just extract the members within the file using the namelist directly: https://docs.python.org/3/library/zipfile.html#zipfile.ZipFile.namelist One could also check our current member list against that namelist and skip the ones that aren't present.
I guess that the original idea of extracing these file explicitly is so that if the zipfile doesn't have the critical files it raises an error rather than carries on regardless.
should we split into "essential (.shp .dbx .shx)" and the rest. Then raise an error if the essential are not there and warn if any others are missing? cpg should certainly be optional based on:
Some info on extensions is here: https://gisgeography.com/arcgis-shapefile-files-types-extensions/
CPG files are optional plain text files that describe the encoding applied to create the shapefile. If your shapefile doesn’t have a .cpg file, then it has the system default encoding.
