atlite
atlite copied to clipboard
PyPSA
Impossible to download Earth-scale data
Description
While improving PyPSA-Africa towards PyPSA-Earth, I've noticed that when I try to create a cutout using atlite, atlite returns an error because copernicus is not able to convert grib files into netCDF format, as requested by the atlite package, and the workflow crashes.
Expected Behavior
Atlite shall be able to derive the desiderable cutout for the Earth. The error message below can be reproduced by using the following code:
import atlite
cutout = atlite.Cutout(path="world_2013.nc",
module="era5",
x=slice(-179.4, 179.7),
y=slice(-59.1, 87.3),
time="2013"
)
cutout.prepare()
Actual Behavior
The workflow stops because Atlite is not able to successfully download the data from copernicus. The raw grib files seem to be procuced but they cannot be successfully converted into netCDF format, hence the procedure stops. When creating the cutout for the world exactly :"area": [87.3, -179.4, -59.1, 179.7] in copernicus.
Error Message
The request you have submitted is not valid
Reason: grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints Cannot create netCDF classic format, dataset is too large! Try splitting the input GRIB(s). grib_to_netcdf: Version 2.24.2 grib_to_netcdf: Processing input file '/cache/tmp/8599e6f4-dc3e-423e-a66e-72e4d44ac365-adaptor.mars.internal-1645467199.6672807-10228-8-tmp.grib'. grib_to_netcdf: Found 17520 GRIB fields in 1 file. grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate grib_to_netcdf: Creating netCDF file '/cache/data3/adaptor.mars.internal-1645468582.836742-10228-15-8599e6f4-dc3e-423e-a66e-72e4d44ac365.nc' grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $ grib_to_netcdf: Creating large (64 bit) file format. grib_to_netcdf: Defining variable 't2m'. grib_to_netcdf: Defining variable 'stl4'.
Your Environment
- The
atliteversion used:0.2.5 - How you installed
atlite(conda,piporgithub):conda - Operating System:
Linux - My environment:
CentOS Linux(output of `conda list`)
``` # Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge affine 2.3.0 py_0 conda-forge alsa-lib 1.2.3 h516909a_0 conda-forge ampl-mp 3.1.0 h2cc385e_1005 conda-forge amply 0.1.4 py_0 conda-forge anyio 3.4.0 py39hf3d152e_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge argon2-cffi 21.1.0 py39h3811e60_2 conda-forge async_generator 1.10 py_0 conda-forge atlite 0.2.5 pyhd8ed1ab_0 conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge babel 2.9.1 pyh44b312d_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.10.0 pyha770c72_0 conda-forge bleach 4.1.0 pyhd8ed1ab_0 conda-forge blosc 1.21.0 h9c3ff4c_0 conda-forge bokeh 2.4.2 py39hf3d152e_0 conda-forge boost-cpp 1.74.0 h312852a_4 conda-forge bottleneck 1.3.2 py39hce5d2b2_5 conda-forge branca 0.4.2 pyhd8ed1ab_0 conda-forge brotli 1.0.9 h7f98852_6 conda-forge brotli-bin 1.0.9 h7f98852_6 conda-forge brotlipy 0.7.0 py39h3811e60_1003 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge c-ares 1.18.1 h7f98852_0 conda-forge ca-certificates 2021.10.8 ha878542_0 conda-forge cairo 1.16.0 h6cf1ce9_1008 conda-forge cartopy 0.20.1 py39ha48bd0d_1 conda-forge cdsapi 0.5.1 pyhd8ed1ab_0 conda-forge certifi 2021.10.8 py39hf3d152e_1 conda-forge cffi 1.15.0 py39h4bc2ebd_0 conda-forge cfgv 3.3.1 pyhd8ed1ab_0 conda-forge cfitsio 3.470 hb418390_7 conda-forge cftime 1.5.1.1 py39hce5d2b2_1 conda-forge charset-normalizer 2.0.8 pyhd8ed1ab_0 conda-forge click 7.1.2 pyh9f0ad1d_0 conda-forge click-plugins 1.1.1 py_0 conda-forge cligj 0.7.2 pyhd8ed1ab_1 conda-forge cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge coincbc 2.10.5 hcee13e7_1 conda-forge colorama 0.4.4 pyh9f0ad1d_0 conda-forge colorcet 2.0.6 pyhd8ed1ab_0 conda-forge conda 4.11.0 py39hf3d152e_0 conda-forge conda-package-handling 1.7.3 py39h3811e60_1 conda-forge configargparse 1.5.3 pyhd8ed1ab_0 conda-forge connection_pool 0.0.3 pyhd3deb0d_0 conda-forge contextily 1.2.0 pyhd8ed1ab_0 conda-forge country_converter 0.7.4 pyhd8ed1ab_0 conda-forge countrycode 0.2 pypi_0 pypi cryptography 36.0.0 py39h95dcef6_0 conda-forge curl 7.80.0 h2574ce0_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cytoolz 0.11.2 py39h3811e60_1 conda-forge dask 2021.3.1 pyhd8ed1ab_0 conda-forge dask-core 2021.3.1 pyhd8ed1ab_0 conda-forge datashader 0.13.0 pyh6c4a22f_0 conda-forge datashape 0.5.4 py_1 conda-forge datrie 0.8.2 py39h3811e60_3 conda-forge dbus 1.13.6 h48d8840_2 conda-forge debugpy 1.5.1 py39he80948d_0 conda-forge decorator 5.1.0 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge deprecation 2.1.0 pyh9f0ad1d_0 conda-forge descartes 1.1.0 py_4 conda-forge distlib 0.3.3 pyhd8ed1ab_0 conda-forge distributed 2021.4.1 py39hf3d152e_1 conda-forge docutils 0.18.1 py39hf3d152e_0 conda-forge editdistance-s 1.0.0 py39h1a9c180_2 conda-forge entrypoints 0.3 pyhd8ed1ab_1003 conda-forge entsoe-py 0.3.7 pyhd8ed1ab_0 conda-forge esy-osm-pbf 0.0.1 pypi_0 pypi esy-osmfilter 1.0.11 pypi_0 pypi et_xmlfile 1.0.1 py_1001 conda-forge expat 2.4.1 h9c3ff4c_0 conda-forge filelock 3.4.0 pyhd8ed1ab_0 conda-forge fiona 1.8.20 py39h427c1bf_2 conda-forge folium 0.12.1.post1 pyhd8ed1ab_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.13.1 hba837de_1005 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.28.2 py39h3811e60_0 conda-forge freetype 2.10.4 h0708190_1 conda-forge freexl 1.0.6 h7f98852_0 conda-forge fsspec 2021.11.0 pyhd8ed1ab_0 conda-forge gdal 3.3.3 py39h0494519_0 conda-forge gdrive 2.1.0 haa36a5b_0 conda-forge geographiclib 1.52 pyhd8ed1ab_0 conda-forge geopandas 0.10.2 pyhd8ed1ab_0 conda-forge geopandas-base 0.10.2 pyha770c72_0 conda-forge geoplot 0.4.4 pyhd8ed1ab_0 conda-forge geopy 2.2.0 pyhd8ed1ab_0 conda-forge geos 3.9.1 h9c3ff4c_2 conda-forge geotiff 1.7.0 hcfb7246_3 conda-forge geoviews 1.9.2 pyhd8ed1ab_0 conda-forge geoviews-core 1.9.2 pyha770c72_0 conda-forge gettext 0.19.8.1 h73d1719_1008 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge gitdb 4.0.9 pyhd8ed1ab_0 conda-forge gitpython 3.1.24 pyhd8ed1ab_0 conda-forge glib 2.70.1 h780b84a_0 conda-forge glib-tools 2.70.1 h780b84a_0 conda-forge glpk 5.0 h445213a_0 conda-forge gmp 6.2.1 h58526e2_0 conda-forge googledrivedownloader 0.4 pyhd3deb0d_1 conda-forge graphite2 1.3.13 h58526e2_1001 conda-forge gst-plugins-base 1.18.5 hf529b03_2 conda-forge gstreamer 1.18.5 h9f60fe5_2 conda-forge harfbuzz 2.9.1 h83ec7ef_1 conda-forge hdf4 4.2.15 h10796ff_3 conda-forge hdf5 1.12.1 nompi_h2750804_102 conda-forge heapdict 1.0.1 py_0 conda-forge holoviews 1.14.6 pyhd8ed1ab_0 conda-forge hvplot 0.7.3 pyh6c4a22f_0 conda-forge icu 68.2 h9c3ff4c_0 conda-forge identify 2.3.7 pyhd8ed1ab_0 conda-forge idna 3.1 pyhd3deb0d_0 conda-forge importlib-metadata 4.8.2 py39hf3d152e_0 conda-forge importlib_metadata 4.8.2 hd8ed1ab_0 conda-forge importlib_resources 5.4.0 pyhd8ed1ab_0 conda-forge iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge ipopt 3.14.4 h7ede334_0 conda-forge ipykernel 6.5.0 py39hef51801_1 conda-forge ipython 7.29.0 py39hef51801_2 conda-forge ipython_genutils 0.2.0 py_1 conda-forge jbig 2.1 h7f98852_2003 conda-forge jedi 0.18.1 py39hf3d152e_0 conda-forge jinja2 3.0.3 pyhd8ed1ab_0 conda-forge joblib 1.1.0 pyhd8ed1ab_0 conda-forge jpeg 9d h36c2ea0_0 conda-forge json-c 0.15 h98cffda_0 conda-forge json5 0.9.5 pyh9f0ad1d_0 conda-forge jsonschema 4.2.1 pyhd8ed1ab_0 conda-forge jupyter_client 7.1.0 pyhd8ed1ab_0 conda-forge jupyter_core 4.9.1 py39hf3d152e_1 conda-forge jupyter_server 1.12.0 pyhd8ed1ab_0 conda-forge jupyterlab 3.2.4 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge jupyterlab_server 2.8.2 pyhd8ed1ab_0 conda-forge kealib 1.4.14 h87e4c3c_3 conda-forge kiwisolver 1.3.2 py39h1a9c180_1 conda-forge krb5 1.19.2 hcc1bbae_3 conda-forge lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge lerc 3.0 h9c3ff4c_0 conda-forge libarchive 3.5.2 hccf745f_1 conda-forge libblas 3.9.0 12_linux64_openblas conda-forge libbrotlicommon 1.0.9 h7f98852_6 conda-forge libbrotlidec 1.0.9 h7f98852_6 conda-forge libbrotlienc 1.0.9 h7f98852_6 conda-forge libcblas 3.9.0 12_linux64_openblas conda-forge libclang 11.1.0 default_ha53f305_1 conda-forge libcurl 7.80.0 h2574ce0_0 conda-forge libdap4 3.20.6 hd7c4107_2 conda-forge libdeflate 1.8 h7f98852_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 h516909a_1 conda-forge libevent 2.1.10 h9b69904_4 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 11.2.0 h1d223b6_11 conda-forge libgdal 3.3.3 h356f897_0 conda-forge libgfortran-ng 11.2.0 h69a702a_11 conda-forge libgfortran5 11.2.0 h5c6108e_11 conda-forge libglib 2.70.1 h174f98d_0 conda-forge libgomp 11.2.0 h1d223b6_11 conda-forge libiconv 1.16 h516909a_0 conda-forge libkml 1.3.0 h238a007_1014 conda-forge liblapack 3.9.0 12_linux64_openblas conda-forge libllvm10 10.0.1 he513fc3_3 conda-forge libllvm11 11.1.0 hf817b99_2 conda-forge libmamba 0.18.2 h3985d26_0 conda-forge libmambapy 0.18.2 py39h8bfa403_0 conda-forge libnetcdf 4.8.1 nompi_hb3fd0d9_101 conda-forge libnghttp2 1.43.0 h812cca2_1 conda-forge libnsl 2.0.0 h7f98852_0 conda-forge libogg 1.3.4 h7f98852_1 conda-forge libopenblas 0.3.18 pthreads_h8fe5266_0 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libpq 13.5 hd57d9b9_0 conda-forge librttopo 1.1.0 h1185371_6 conda-forge libsodium 1.0.18 h36c2ea0_1 conda-forge libsolv 0.7.19 h780b84a_5 conda-forge libspatialindex 1.9.3 h9c3ff4c_4 conda-forge libspatialite 5.0.1 h8796b1e_9 conda-forge libssh2 1.10.0 ha56f1ee_2 conda-forge libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge libtiff 4.3.0 h6f004c6_2 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp-base 1.2.1 h7f98852_0 conda-forge libxcb 1.13 h7f98852_1004 conda-forge libxkbcommon 1.0.3 he3ba5ed_0 conda-forge libxml2 2.9.12 h72842e0_0 conda-forge libxslt 1.1.33 h15afd5d_2 conda-forge libzip 1.8.0 h4de3113_1 conda-forge libzlib 1.2.11 h36c2ea0_1013 conda-forge llvmlite 0.36.0 py39h1bbdace_0 conda-forge locket 0.2.0 py_2 conda-forge lxml 4.6.4 py39h107f48f_0 conda-forge lz4-c 1.9.3 h9c3ff4c_1 conda-forge lzo 2.10 h516909a_1000 conda-forge mamba 0.18.2 py39hfa8f2c8_0 conda-forge mapclassify 2.4.3 pyhd8ed1ab_0 conda-forge markdown 3.3.6 pyhd8ed1ab_0 conda-forge markupsafe 2.0.1 py39h3811e60_1 conda-forge matplotlib 3.5.0 py39hf3d152e_0 conda-forge matplotlib-base 3.5.0 py39h2fa2bec_0 conda-forge matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge memory_profiler 0.58.0 py_0 conda-forge mercantile 1.2.1 pyhd8ed1ab_0 conda-forge metis 5.1.0 h58526e2_1006 conda-forge mistune 0.8.4 py39h3811e60_1005 conda-forge mock 4.0.3 py39hf3d152e_2 conda-forge more-itertools 8.12.0 pyhd8ed1ab_0 conda-forge msgpack-python 1.0.3 py39h1a9c180_0 conda-forge multipledispatch 0.6.0 py_0 conda-forge mumps-include 5.2.1 ha770c72_10 conda-forge mumps-seq 5.2.1 h47a8eb5_10 conda-forge munch 2.5.0 py_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.27 ha770c72_1 conda-forge mysql-libs 8.0.27 hfa10184_1 conda-forge nbclassic 0.3.4 pyhd8ed1ab_0 conda-forge nbclient 0.5.9 pyhd8ed1ab_0 conda-forge nbconvert 6.3.0 py39hf3d152e_1 conda-forge nbformat 5.1.3 pyhd8ed1ab_0 conda-forge ncurses 6.2 h58526e2_4 conda-forge nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge netcdf4 1.5.8 nompi_py39h64b754b_101 conda-forge networkx 2.6.3 pyhd8ed1ab_1 conda-forge nodeenv 1.6.0 pyhd8ed1ab_0 conda-forge nomkl 1.0 h5ca1d4c_0 conda-forge notebook 6.4.6 pyha770c72_0 conda-forge nspr 4.32 h9c3ff4c_1 conda-forge nss 3.72 hb5efdd6_0 conda-forge numba 0.53.1 py39h56b8d98_1 conda-forge numexpr 2.7.3 py39hbd72853_102 conda-forge numpy 1.21.4 py39hdbf815f_0 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjdk 11.0.9.1 h5cc2fde_1 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openpyxl 3.0.9 pyhd8ed1ab_0 conda-forge openssl 1.1.1l h7f98852_0 conda-forge packaging 21.3 pyhd8ed1ab_0 conda-forge pandas 1.3.4 py39hde0f152_1 conda-forge pandoc 2.16.2 h7f98852_0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge panel 0.12.4 pyhd8ed1ab_0 conda-forge param 1.12.0 pyh6c4a22f_0 conda-forge parso 0.8.2 pyhd8ed1ab_0 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge patsy 0.5.2 pyhd8ed1ab_0 conda-forge pcre 8.45 h9c3ff4c_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pillow 8.4.0 py39ha612740_0 conda-forge pip 21.3.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge pluggy 1.0.0 py39hf3d152e_2 conda-forge ply 3.11 py_1 conda-forge poppler 21.09.0 ha39eefc_3 conda-forge poppler-data 0.4.11 hd8ed1ab_0 conda-forge postgresql 13.5 h2510834_0 conda-forge powerplantmatching 0.4.8 pyhd8ed1ab_0 conda-forge pre-commit 2.15.0 py39hf3d152e_1 conda-forge progressbar2 3.55.0 pyhd8ed1ab_0 conda-forge proj 8.1.1 h277dcde_2 conda-forge prometheus_client 0.12.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.22 pyha770c72_0 conda-forge protobuf 3.19.1 pypi_0 pypi psutil 5.8.0 py39h3811e60_2 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pulp 2.5.1 py39hf3d152e_1 conda-forge py 1.11.0 pyh6c4a22f_0 conda-forge pybind11-abi 4 hd8ed1ab_3 conda-forge pycosat 0.6.3 py39h3811e60_1009 conda-forge pycountry 20.7.3 pyh9f0ad1d_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pyct 0.4.6 py_0 conda-forge pyct-core 0.4.6 py_0 conda-forge pygments 2.10.0 pyhd8ed1ab_0 conda-forge pyomo 6.2 py39he80948d_0 conda-forge pyopenssl 21.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.6 pyhd8ed1ab_0 conda-forge pyproj 3.2.1 py39ha81a305_2 conda-forge pypsa 0.18.1 pyhd8ed1ab_0 conda-forge pyqt 5.12.3 py39hf3d152e_8 conda-forge pyqt-impl 5.12.3 py39hde8b62d_8 conda-forge pyqt5-sip 4.19.18 py39he80948d_8 conda-forge pyqtchart 5.12 py39h0fcd23e_8 conda-forge pyqtwebengine 5.12.1 py39h0fcd23e_8 conda-forge pyrsistent 0.18.0 py39h3811e60_0 conda-forge pyshp 2.1.3 pyh44b312d_0 conda-forge pysocks 1.7.1 py39hf3d152e_4 conda-forge pytables 3.6.1 py39h2669a42_4 conda-forge pytest 6.2.5 py39hf3d152e_1 conda-forge python 3.9.7 hb7a2778_3_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-utils 2.5.6 pyh44b312d_0 conda-forge python_abi 3.9 2_cp39 conda-forge pytz 2021.3 pyhd8ed1ab_0 conda-forge pyviz_comms 2.1.0 pyhd8ed1ab_0 conda-forge pyyaml 6.0 py39h3811e60_3 conda-forge pyzmq 22.3.0 py39h37b5a0c_1 conda-forge qt 5.12.9 hda022c4_4 conda-forge rasterio 1.2.10 py39hb37810a_0 conda-forge ratelimiter 1.2.0 py_1002 conda-forge readline 8.1 h46c0cb4_0 conda-forge reproc 14.2.3 h7f98852_0 conda-forge reproc-cpp 14.2.3 h9c3ff4c_0 conda-forge requests 2.26.0 pyhd8ed1ab_1 conda-forge rioxarray 0.9.0 pyhd8ed1ab_0 conda-forge rtree 0.9.7 py39hb102c33_3 conda-forge ruamel_yaml 0.15.80 py39h3811e60_1006 conda-forge scikit-learn 1.0.1 py39h4dfa638_2 conda-forge scipy 1.7.3 py39hee8e79c_0 conda-forge scotch 6.0.9 h3858553_1 conda-forge seaborn 0.11.2 hd8ed1ab_0 conda-forge seaborn-base 0.11.2 pyhd8ed1ab_0 conda-forge send2trash 1.8.0 pyhd8ed1ab_0 conda-forge setuptools 59.2.0 py39hf3d152e_0 conda-forge shapely 1.8rc1 py39h2226dc9_0 conda-forge/label/shapely_dev six 1.16.0 pyh6c4a22f_0 conda-forge sklearn 0.0 pypi_0 pypi smart_open 5.2.1 pyhd8ed1ab_0 conda-forge smmap 3.0.5 pyh44b312d_0 conda-forge snakemake-minimal 6.11.0 pyhdfd78af_0 bioconda sniffio 1.2.0 py39hf3d152e_2 conda-forge snuggs 1.4.7 py_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3 pyhd8ed1ab_0 conda-forge sqlite 3.36.0 h9cd32fc_2 conda-forge statsmodels 0.13.1 py39hce5d2b2_0 conda-forge stopit 1.1.2 py_0 conda-forge tabulate 0.8.9 pyhd8ed1ab_0 conda-forge tblib 1.7.0 pyhd8ed1ab_0 conda-forge terminado 0.12.1 py39hf3d152e_1 conda-forge testpath 0.5.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.0.0 pyh8a188c0_0 conda-forge tiledb 2.3.4 he87e0bf_0 conda-forge tk 8.6.11 h27826a3_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge toolz 0.11.2 pyhd8ed1ab_0 conda-forge toposort 1.7 pyhd8ed1ab_0 conda-forge tornado 6.1 py39h3811e60_2 conda-forge tqdm 4.62.3 pyhd8ed1ab_0 conda-forge traitlets 5.1.1 pyhd8ed1ab_0 conda-forge tsam 1.1.2 pypi_0 pypi typing_extensions 4.0.0 pyha770c72_0 conda-forge tzcode 2021e h7f98852_0 conda-forge tzdata 2021e he74cb21_0 conda-forge urllib3 1.26.7 pyhd8ed1ab_0 conda-forge virtualenv 20.4.7 py39hf3d152e_1 conda-forge vresutils 0.3.1 pypi_0 pypi wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 1.2.1 py39hf3d152e_0 conda-forge wheel 0.37.0 pyhd8ed1ab_1 conda-forge wrapt 1.13.3 py39h3811e60_1 conda-forge xarray 0.20.2.dev19+g21e84842 pypi_0 pypi xerces-c 3.2.3 h9d8b166_3 conda-forge xlrd 2.0.1 pyhd8ed1ab_3 conda-forge xorg-fixesproto 5.0 h7f98852_1002 conda-forge xorg-inputproto 2.3.2 h7f98852_1002 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h7f98852_0 conda-forge xorg-libsm 1.2.3 hd9c2040_1000 conda-forge xorg-libx11 1.7.2 h7f98852_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h7f98852_1 conda-forge xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge xorg-libxi 1.7.10 h7f98852_0 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-libxtst 1.2.3 h7f98852_1002 conda-forge xorg-recordproto 1.14.2 h7f98852_1002 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xyzservices 2021.11.0 pyhd8ed1ab_0 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h516909a_0 conda-forge yaml-cpp 0.6.3 he1b5a44_4 conda-forge zeromq 4.3.4 h9c3ff4c_1 conda-forge zict 2.0.0 py_0 conda-forge zipp 3.6.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h36c2ea0_1013 conda-forge zstd 1.5.0 ha95c52a_0 conda-forge ```
Can reproduce. From a first glance this might be an issue on the CDS API side. classic netCDF is limited in size to ~2 GB, maybe the individual requests are exceeding this size and the api does not internally switch to netCDF v4 (which allows for larger sizes)? We'd need to investigate a bit more, maybe someone else has a clearer view?
Full stack trace:
2022-02-22 08:15:33,413 INFO Requesting data for feature influx...
2022-02-22 08:23:52,061 INFO CDS: Downloading variables
* geopotential (2013)
2022-02-22 08:59:55,175 ERROR Message: the request you have submitted is not valid
2022-02-22 08:59:55,176 ERROR Reason:
grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints
Cannot create netCDF classic format, dataset is too large!
Try splitting the input GRIB(s).
grib_to_netcdf: Version 2.24.2
grib_to_netcdf: Processing input file '/cache/tmp/74fcce81-8102-4388-b11b-f71e8c4d56a3-adaptor.mars.internal-1645514525.1224709-4276-8-tmp.grib'.
grib_to_netcdf: Found 17520 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file '/cache/data5/adaptor.mars.internal-1645515842.3469903-4276-6-74fcce81-8102-4388-b11b-f71e8c4d56a3.nc'
grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $
grib_to_netcdf: Creating large (64 bit) file format.
grib_to_netcdf: Defining variable 't2m'.
grib_to_netcdf: Defining variable 'stl4'.
2022-02-22 08:59:55,177 ERROR Traceback (most recent call last):
2022-02-22 08:59:55,178 ERROR File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 55, in handle_request
2022-02-22 08:59:55,179 ERROR result = cached(context.method, proc, context, context.args, context.kwargs)
2022-02-22 08:59:55,179 ERROR File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
2022-02-22 08:59:55,180 ERROR result = proc(context, *context.args, **context.kwargs)
2022-02-22 08:59:55,180 ERROR File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 118, in __call__
2022-02-22 08:59:55,180 ERROR return p(*args, **kwargs)
2022-02-22 08:59:55,181 ERROR File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 59, in __call__
2022-02-22 08:59:55,181 ERROR return self.proc(context, *args, **kwargs)
2022-02-22 08:59:55,181 ERROR File "/home/cds/cdsservices/services/mars/mars.py", line 47, in internal
2022-02-22 08:59:55,181 ERROR return mars(context, request, **kwargs)
2022-02-22 08:59:55,182 ERROR File "/home/cds/cdsservices/services/mars/mars.py", line 25, in mars
2022-02-22 08:59:55,182 ERROR grib_to_netcdf(context, requests, info)
2022-02-22 08:59:55,182 ERROR File "/home/cds/cdsservices/services/mars/grib_to_netcdf.py", line 42, in grib_to_netcdf
2022-02-22 08:59:55,183 ERROR context.run_command(*cmd, exception=NetcdfException)
2022-02-22 08:59:55,183 ERROR File "/opt/cdstoolbox/cdscompute/cdscompute/context.py", line 209, in run_command
2022-02-22 08:59:55,184 ERROR raise exception(call, proc.returncode, output)
2022-02-22 08:59:55,184 ERROR home.cds.cdsservices.services.mars.__init__.py.exceptions.NetcdfException:
2022-02-22 08:59:55,185 ERROR grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
Input In [6], in <module>
----> 1 cutout.prepare()
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/data.py:102, in maybe_remove_tmpdir.<locals>.wrapper(*args, **kwargs)
100 kwargs["tmpdir"] = mkdtemp()
101 try:
--> 102 res = func(*args, **kwargs)
103 finally:
104 rmtree(kwargs["tmpdir"])
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/data.py:164, in cutout_prepare(cutout, features, tmpdir, overwrite)
162 logger.info(f"Calculating and writing with module {module}:")
163 missing_features = missing_vars.index.unique("feature")
--> 164 ds = get_features(cutout, module, missing_features, tmpdir=tmpdir)
165 prepared |= set(missing_features)
167 cutout.data.attrs.update(dict(prepared_features=list(prepared)))
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/data.py:46, in get_features(cutout, module, features, tmpdir)
41 feature_data = delayed(get_data)(
42 cutout, feature, tmpdir=tmpdir, lock=lock, **parameters
43 )
44 datasets.append(feature_data)
---> 46 datasets = compute(*datasets)
48 ds = xr.merge(datasets, compat="equals")
49 for v in ds:
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/base.py:571, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
568 keys.append(x.__dask_keys__())
569 postcomputes.append(x.__dask_postcompute__())
--> 571 results = schedule(dsk, keys, **kwargs)
572 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/threaded.py:79, in get(dsk, result, cache, num_workers, pool, **kwargs)
76 elif isinstance(pool, multiprocessing.pool.Pool):
77 pool = MultiprocessingPoolExecutor(pool)
---> 79 results = get_async(
80 pool.submit,
81 pool._max_workers,
82 dsk,
83 result,
84 cache=cache,
85 get_id=_thread_get_id,
86 pack_exception=pack_exception,
87 **kwargs,
88 )
90 # Cleanup pools associated to dead threads
91 with pools_lock:
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/local.py:507, in get_async(submit, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, chunksize, **kwargs)
505 _execute_task(task, data) # Re-execute locally
506 else:
--> 507 raise_exception(exc, tb)
508 res, worker_id = loads(res_info)
509 state["cache"][key] = res
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/local.py:315, in reraise(exc, tb)
313 if exc.__traceback__ is not tb:
314 raise exc.with_traceback(tb)
--> 315 raise exc
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/local.py:220, in execute_task(key, task_info, dumps, loads, get_id, pack_exception)
218 try:
219 task, data = loads(task_info)
--> 220 result = _execute_task(task, data)
221 id = get_id()
222 result = dumps((result, id))
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/core.py:119, in _execute_task(arg, cache, dsk)
115 func, args = arg[0], arg[1:]
116 # Note: Don't assign the subtask results to a variable. numpy detects
117 # temporaries by their reference count and can execute certain
118 # operations in-place.
--> 119 return func(*(_execute_task(a, cache) for a in args))
120 elif not ishashable(arg):
121 return arg
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/utils.py:40, in apply(func, args, kwargs)
38 def apply(func, args, kwargs=None):
39 if kwargs:
---> 40 return func(*args, **kwargs)
41 else:
42 return func(*args)
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:378, in get_data(cutout, feature, tmpdir, lock, **creation_parameters)
374 return retrieve_once(retrieval_times(coords, static=True)).squeeze()
376 datasets = map(retrieve_once, retrieval_times(coords))
--> 378 return xr.concat(datasets, dim="time").sel(time=coords["time"])
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/xarray/core/concat.py:220, in concat(objs, dim, data_vars, coords, compat, positions, fill_value, join, combine_attrs)
217 from .dataset import Dataset
219 try:
--> 220 first_obj, objs = utils.peek_at(objs)
221 except StopIteration:
222 raise ValueError("must supply at least one object to concatenate")
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/xarray/core/utils.py:196, in peek_at(iterable)
192 """Returns the first value from iterable, as well as a new iterator with
193 the same content as the original iterable
194 """
195 gen = iter(iterable)
--> 196 peek = next(gen)
197 return peek, itertools.chain([peek], gen)
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:368, in get_data.<locals>.retrieve_once(time)
367 def retrieve_once(time):
--> 368 ds = func({**retrieval_params, **time})
369 if sanitize and sanitize_func is not None:
370 ds = sanitize_func(ds)
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:192, in get_data_temperature(retrieval_params)
190 def get_data_temperature(retrieval_params):
191 """Get wind temperature for given retrieval parameters."""
--> 192 ds = retrieve_data(
193 variable=["2m_temperature", "soil_temperature_level_4"], **retrieval_params
194 )
196 ds = _rename_and_clean_coords(ds)
197 ds = ds.rename({"t2m": "temperature", "stl4": "soil temperature"})
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:301, in retrieve_data(product, chunks, tmpdir, lock, **updates)
294 assert {"year", "month", "variable"}.issubset(
295 request
296 ), "Need to specify at least 'variable', 'year' and 'month'"
298 client = cdsapi.Client(
299 info_callback=logger.debug, debug=logging.DEBUG >= logging.root.level
300 )
--> 301 result = client.retrieve(product, request)
303 if lock is None:
304 lock = nullcontext()
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/cdsapi/api.py:348, in Client.retrieve(self, name, request, target)
347 def retrieve(self, name, request, target=None):
--> 348 result = self._api("%s/resources/%s" % (self.url, name), request, "POST")
349 if target is not None:
350 result.download(target)
File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/cdsapi/api.py:506, in Client._api(self, url, request, method)
504 break
505 self.error(" %s", n)
--> 506 raise Exception(
507 "%s. %s."
508 % (reply["error"].get("message"), reply["error"].get("reason"))
509 )
511 raise Exception("Unknown API state [%s]" % (reply["state"],))
Exception: the request you have submitted is not valid.
grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints
Cannot create netCDF classic format, dataset is too large!
Try splitting the input GRIB(s).
grib_to_netcdf: Version 2.24.2
grib_to_netcdf: Processing input file '/cache/tmp/74fcce81-8102-4388-b11b-f71e8c4d56a3-adaptor.mars.internal-1645514525.1224709-4276-8-tmp.grib'.
grib_to_netcdf: Found 17520 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file '/cache/data5/adaptor.mars.internal-1645515842.3469903-4276-6-74fcce81-8102-4388-b11b-f71e8c4d56a3.nc'
grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $
grib_to_netcdf: Creating large (64 bit) file format.
grib_to_netcdf: Defining variable 't2m'.
grib_to_netcdf: Defining variable 'stl4'.
.
I'm wondering whether atlite needs to download copernicus data as NC format; the nc extraction is also experimental in copernicus. A possible solution could be to use the raw grib format when downloading data from copernicus, xarray should be able to load such data accordingly, yet I'm not sure whether the loading may need some adaptations.
https://stackoverflow.com/questions/64937550/converting-grib-to-netcdf4
Update: maybe not possible; see https://confluence.ecmwf.int/display/CKB/Climate+Data+Store+%28CDS%29+API+Keywords