OpenDrift
NetCDF > 1.6.1 does not use .netrc/.dodsrc: Trouble accessing CMEMS sources
Hi,
I installed opendrift folowing instructions you provide. When using a conda environment I am not able to reach CMEMS source, no mather where I add my .dodsrc and .netrc file.
I have no problems using ncdump -h 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i' outside of conda environment but after doing:
conda activate opendrift
and
(opendrift) [root@opendrift-poc ~]# ncdump -h 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'
I get the following error message:
syntax error, unexpected WORD_WORD, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR context: <!DOCTYPE^ html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <head> <title>Login</title> <link href="http://marine.copernicus.eu/favicon.ico" rel="shortcut icon" type="image/x-icon" /> <!-- link rel="icon" type="image/png" href="http://myo-wp.techworks.ie/devel/templates/myocean/favicon.png" /-->> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/content.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/content_cc.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/myocean/screen/patch_content_cc.css" media="all" /> <link rel="stylesheet" type="text/css" href="/cas/newCss/css/cssmanager.php?version=4.0.2-1288885946&files=/cas/newCss/css/myocean/print/print_003.css" media="print" /> <!--[if gte IE 6]><style type="text/css" media="screen">@import 'css/ie_cas.css';</style><![endif]--> <script type="text/javascript" src="js/common_rosters.js"></script> <style type="text/css">input.submit{cursor:pointer}</style> </head> <body id="cas" class="claro" onload="init();" > <div id="white_background_inner_page" style="height:105px;"></div> <div id="col3_content" class="clearfix"> <img src="/css/myocean/print/images/logo_myocean_print.png" width="1" height="1" align="left" id="logoPrint" alt="" /> <div id="breadcrumb"> <a href="http://marine.copernicus.eu/">Home</a> > Login </div> <h1>Welcome to the CMEMS Data Access Portal</h1> <br/> <p>Fill your login/password and click on <strong>Login</strong> to download data. If you are not registred yet click on <a href="http://marine.copernicus.eu/web/56-user-registration-form.php" target="_parent">Register</a></p> <p>If you have trouble logging in, make sure your browser is set to accept cookies.</p> <br/> <form id="authentification" name="com-login" class="fm-v clearfix" action="/cas/login;jsessionid=BA9435DA67876D405E6490F3DFC023A9?service=http://localhost:8080/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i.dds" method="post"> <fieldset class="input"> <label for="username">Username</label> <input id="username" name="username" class="required" tabindex="1" accesskey="i" type="text" value="" size="25" autocomplete="false"/><br /> <label for="password">Password</label> <input id="password" name="password" class="required" tabindex="2" accesskey="m" type="password" value="" size="25" autocomplete="off"/><br /> <input type="hidden" name="lt" value="_c4793F86B-451F-2DD7-ECFA-D395B0B99CF4_k3D0FA0E2-B51A-ACF8-EF1A-B307A76D684E" /> <input type="hidden" name="_eventId" value="submit" /> <div class="authentification_retrait_right"> <a href="http://marine.copernicus.eu/web/56-user-registration-form.php" target="_parent"><img src="/cas/newCss/css/myocean/screen/images/bt_register.gif" alt="Register" /></a> <input class="submit" name="submit" accesskey="l" value=" " type="submit" alt="Login" /> </div> </fieldset> <p> <br/><br/> <i>- You are registered but have forgotten your login/password?</i><br/> <i>- Any question?</i><br/> <b>⇒ CMEMS Service Desk : </b><A HREF="mailto:[email protected]"><b>[email protected]</b></A> </p> </form> <h3>Thank you for using CMEMS products</h3> <br/> <div id="sidebar"> <i>For security reasons, please Exit your web browser when you quit services requiring authentication!</i> </div> </div> </body></html> ncdump: https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i: NetCDF: Access failure
Of course, using python's modules inside the env yields similar results
from opendrift.readers import reader_netCDF_CF_generic
reader_netCDF_CF_generic.Reader('https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i')
results in
Traceback (most recent call last):
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 210, in _acquire_with_cache_info
file = self._cache[self._key]
~~~~~~~~~~~^^^^^^^^^^^
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
value = self._cache[key]
~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '297eccf1-3b01-443c-a779-283fa9800280']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/opendrift/readers/reader_netCDF_CF_generic.py", line 146, in __init__
self.Dataset = xr.open_dataset(filename, decode_times=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/api.py", line 526, in open_dataset
backend_ds = backend.open_dataset(
^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 577, in open_dataset
store = NetCDF4DataStore.open(
^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 382, in open
return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 329, in __init__
self.format = self.ds.data_model
^^^^^^^
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 391, in ds
return self._acquire()
^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 385, in _acquire
with self._manager.acquire_context(needs_lock) as root:
File "/root/miniconda3/envs/opendrift/lib/python3.11/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 198, in acquire_context
file, cached = self._acquire_with_cache_info(needs_lock)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 216, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src/netCDF4/_netCDF4.pyx", line 2449, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 2012, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -77] NetCDF: Access failure: 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/root/miniconda3/envs/opendrift/lib/python3.11/site-packages/opendrift/readers/reader_netCDF_CF_generic.py", line 148, in __init__
raise ValueError(e)
ValueError: [Errno -77] NetCDF: Access failure: 'https://nrt.cmems-du.eu/thredds/dodsC/cmems_mod_glo_phy_anfc_merged-uv_PT1H-i'
The only instalation that worked is the one via a docker file, but it doesn't suit my needs.
my conda environments is as follows:
active environment : opendrift
active env location : /root/miniconda3/envs/opendrift
shell level : 2
user config file : /root/.condarc
populated config files : /root/.condarc
conda version : 23.1.0
conda-build version : not installed
python version : 3.10.9.final.0
virtual packages : __archspec=1=x86_64
__glibc=2.28=0
__linux=4.18.0=0
__unix=0=0
base environment : /root/miniconda3 (writable)
conda av data dir : /root/miniconda3/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/linux-64
https://conda.anaconda.org/conda-forge/noarch
https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /root/miniconda3/pkgs
/root/.conda/pkgs
envs directories : /root/miniconda3/envs
/root/.conda/envs
platform : linux-64
user-agent : conda/23.1.0 requests/2.28.1 CPython/3.10.9 Linux/4.18.0-425.19.2.el8_7.x86_64 rocky/8.7 glibc/2.28
UID:GID : 0:0
netrc file : /root/.netrc
offline mode : False
# conda environments:
#
base /root/miniconda3
opendrift * /root/miniconda3/envs/opendrift
sys.version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GC...
sys.prefix: /root/miniconda3
sys.executable: /root/miniconda3/bin/python
conda location: /root/miniconda3/lib/python3.10/site-packages/conda
conda-build: None
conda-content-trust: /root/miniconda3/bin/conda-content-trust
conda-env: /root/miniconda3/bin/conda-env
user site dirs:
CIO_TEST: <not set>
CONDA_DEFAULT_ENV: opendrift
CONDA_EXE: /root/miniconda3/bin/conda
CONDA_PREFIX: /root/miniconda3/envs/opendrift
CONDA_PREFIX_1: /root/miniconda3
CONDA_PROMPT_MODIFIER: (opendrift)
CONDA_PYTHON_EXE: /root/miniconda3/bin/python
CONDA_ROOT: /root/miniconda3
CONDA_SHLVL: 2
CURL_CA_BUNDLE: <not set>
GDAL_DRIVER_PATH: /root/miniconda3/envs/opendrift/lib/gdalplugins
LD_PRELOAD: <not set>
PATH: /root/miniconda3/envs/opendrift/bin:/root/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
REQUESTS_CA_BUNDLE: <not set>
SSL_CERT_FILE: <not set>
http_proxy: <set>
https_proxy: <set>
Any help would be greatly appreciated, I have run out of ideas already.
Thank you very much
I also get the same problem when updating my environment, and so that netCDF version is 1.6.3 I tried then to downgrade to netCDF 1.6.2 (working on another machine), but same problem. Also downgrading Xarray to 2022.12.0 (working on another machine) did not work. Then not sure how to fix this?
Would it be possible that the problem comes from the netCDF library, and not from the netCDF4 python module?
I'm assuming that because ncdump works outside the conda environment ( netcdf library version 4.7.0 ) but fails inside it (netcdf library version 4.9.2).
Anyway, I will try to test this and report here
I can make this work for me with conda and netcdf4=1.6.1
You are correct, it worked for me too, even with netcdf library version 4.9.2.
Ok, I have now temporarily locked netcdf4 to <=1.6.1 in opendrift environment.yml, until bug is hopefully solved and backported by someone. https://github.com/OpenDrift/opendrift/commit/4626d20ba1fc07b21c71257624f5dd0beaf5f396
If you update to latest, environment.yml contains netcdf4<=1.6.1
You can then do
conda env create -f environment.yml --force
followed by
pip install -e .
to reinstall dependencies
Is it possible to define the directory where the .dodsrc is present? I want to change the directory where these files are present but the opendrift always will try on HOME dir. We can change this behavior?
.dodsrc and .netrc authentication are general, and not a part of OpenDrift.
You can see specifications here: https://docs.unidata.ucar.edu/nug/current/dap2.html