ncdump error: SSL connect, curl, oc_open could not read url

[DanCodigaMWRA]

I'm in a conda environment on Windows and ncdump is giving me a "could not read url" error (oc_open, SSL connect) when accessing a file on a remote server:

(EQ) PS C:\Users\Codiga_D> ncdump -h http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc
Error:curl error: SSL connect error
curl error details:
Warning:oc_open: Could not read url
C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\Library\bin\ncdump.exe: http

Others report success with this operation so the trouble seems to be with my own system. These seem relevant:

(EQ) PS C:\Users\Codiga_D> conda list curl
# packages in environment at C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ:
#
# Name                    Version                   Build  Channel
curl                      7.83.1               h789b8ee_0    conda-forge
libcurl                   7.83.1               h789b8ee_0    conda-forge

(EQ) PS C:\Users\Codiga_D> conda list certifi
# packages in environment at C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ:
#
# Name                    Version                   Build  Channel
ca-certificates           2022.6.15            h5b45459_0    conda-forge
certifi                   2022.6.15        py37h03978a9_0    conda-forge

The error first occurred for me using xarray,

import xarray as xr
url = 'http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc'
ds = xr. open_dataset(url)

but has been shown to occur in netcdf4 and ncdump as well.

At least several months ago I did not have this problem. I don't have details as to all the changes to my system have occurred since then, but one recent change had to do with certificate chains so I was guessing it might be related to that.

These issues might be related? https://github.com/Unidata/netcdf-c/issues/1393 https://github.com/Unidata/netcdf-c/issues/1833 https://github.com/Unidata/netcdf4-python/issues/755 https://github.com/pydata/xarray/issues/4925

Earlier posts I have made on this, with some additional relevant information: https://github.com/pydata/xarray/issues/6766 https://github.com/pydata/xarray/discussions/6742

Thank you in advance.

[ghost]

In the meantime, until this can properly be resolved, a couple thoughts for potential workarounds occurred to me:

• pin my version of netcdf4, currently 1.6.0, to something earlier?
• use the requests package instead? if this might be possible, and someone could suggest syntax using requests to do equivalent of

import xarray as xr
url = 'http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc'
ds = xr. open_dataset(url)

which might not lead to the same error, that would be very much appreciated!

[DennisHeimbigner]

Using the current netcdf-c master, I cannot duplicate the error. Try changing the URL to this form to get some more information:

http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc#log&show=fetch

[ghost]

Update: I am pretty certain now that this issue is related to my local certification chains here. So I am going to close it for now. If I sort out the certification trouble and this issue persists, I will reopen it.

And thank you @DennisHeimbigner for the suggestion. I tried it various ways but didn't seem to get much new information. Here are the results:

• Using ncdump from console:

P:\>conda activate EQ
(EQ) P:\>ncdump -h http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc#log&show=fetch
Error:curl error: SSL connect error
curl error details:
Warning:oc_open: Could not read url
ncdump: http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc#log: NetCDF: I/O failure
'show' is not recognized as an internal or external command,
operable program or batch file.

• Using xarray in conda env:

import xarray as xr
url = 'http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc#log&show=fetch'
ds = xr.open_dataset(url)

Note:Caching=1
Note:fetch: http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc.dds
Traceback (most recent call last):

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\site-packages\xarray\backends\file_manager.py", line 199, in _acquire_with_cache_info
    file = self._cache[self._key]

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\site-packages\xarray\backends\lru_cache.py", line 53, in __getitem__
    value = self._cache[key]

KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc#log&show=fetch',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]


During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "C:\Users\Codiga_D\AppData\Local\Temp\ipykernel_7324\1695649982.py", line 1, in <module>
    ds = xr.open_dataset(url)

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\site-packages\xarray\backends\api.py", line 499, in open_dataset
    **kwargs,

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\site-packages\xarray\backends\netCDF4_.py", line 559, in open_dataset
    autoclose=autoclose,

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\site-packages\xarray\backends\netCDF4_.py", line 379, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\site-packages\xarray\backends\netCDF4_.py", line 327, in __init__
    self.format = self.ds.data_model

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\site-packages\xarray\backends\netCDF4_.py", line 388, in ds
    return self._acquire()

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\site-packages\xarray\backends\netCDF4_.py", line 382, in _acquire
    with self._manager.acquire_context(needs_lock) as root:

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\contextlib.py", line 112, in __enter__
    return next(self.gen)

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\site-packages\xarray\backends\file_manager.py", line 187, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)

  File "C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\lib\site-packages\xarray\backends\file_manager.py", line 205, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)

  File "src\netCDF4\_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__

  File "src\netCDF4\_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success

OSError: [Errno -68] NetCDF: I/O failure: b'http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc#log&show=fetch'


Error:curl error: SSL connect error
curl error details: 
Warning:oc_open: Could not read url
Note:fetch complete.

[ghost]

Reopening this. I got help sorting out some certification issues so those are resolved, but this issue is still a problem. We think it may be related to certificate chains.

How can I check which certificate file/chain is being used by ncdump? What about curl?

It may be that I need to configure ncdump and/or curl to use a certain certificate file (different from the ones currently being used) and/or append to the certificate file it/they are currently using.

[DennisHeimbigner]

First review the relevant curl documentation and see if anything strikes you as relevant to your situation.

• https://curl.se/libcurl/c/CURLOPT_CAINFO.html
• https://curl.se/libcurl/c/CURLOPT_CAPATH.html

[DennisHeimbigner]

Second, you can force set CAINFO and/or CAPATH by creating the file ~/.ncrc and putting either or both of these lines in it. This assumes a non-windows environment.

• HTTP.SSL.CAINFO=<value>
• HTTP.SSL,CAPATH=<value>

[ghost]

Will look in to this. Thanks for the quick reply Dennis. I am on windows though.

[DennisHeimbigner]

Ok, then you need to put the .ncrc file in %USERPROFILE% instead of $HOME.

[d70-t]

I suspect it's something related to how netCDF-c is compiled (in particular within the netCDF-python release). I just did three experiments on a freshly started ubuntu:20.04 docker image:

• using python3 -m pip install netcdf4==1.6.0 does not work (with Warning:oc_open: Could not read url). It comes with netCDF-c @ 4.9.0
• using python3 -m pip install "netcdf4<1.6.0" resolves to 1.5.8 and does work. It comes with netCDF-c @ 4.7.4
• installing netCDF-c @ 4.9.0 from source (without using Python) does work as well

So it's probably not a general issues with the version of netCDF-c, but maybe related to the build process. I however didn't figure out from where the netCDF-c which is packaged with Python comes from and how it is built....

[ghost]

@DennisHeimbigner I made a .ncrc file with HTTP.SSL.CAPATH in it, and put it in to C:\Users\Codiga_D (this is the correct %USERPROFILE% I hope?), but I still get the same error from ncdump. However, I am unsure of the syntax within the .ncrc file (quote/unquote path? forward/backward slashes?)... here is what I used (single quotes and backslashes):

HTTP.SSL.CAPATH='C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\Lib\site-packages\certifi\cacert.pem'

Guidance?

[ghost]

@d70-t I am in a conda env (not using pip at all) on windows, so I'm not sure if your tests are a close match to my case... but just to try something I pinned my netcdf4 to 1.5.8 (it was 1.6.0) and that did not solve the problem either.

[DennisHeimbigner]

Backward slash should be ok, but do not quote the value (i.e text after the equal sign).

[ghost]

Ok I removed the quotes. Error persists. Not 100% sure the .ncrc file is being used-- is there any kind of test I could do, to confirm that it is?

[DennisHeimbigner]

I was looking around and found this:

The CURLOPT_CAPATH function apparently does not work in Windows due to some limitation in openssl.

So try changing HTTP.SSL.CAPATH=... to HTTP.SSL.CAINFO=... but keeping the same value.

[DennisHeimbigner]

Also, you may be able to test .ncrc is being used by adding this line:

HTTP.VERBOSE=1

If you see a bunch of curl debug output, then .ncrc is being read.

[ghost]

Thanks Dennis. Tried that too (CAINFO instead of CAPATH), still no joy. On the bright side, including HTTP.VERBOSE=1 did cause additional messages to appear-- so that confirms the .ncrc file is being used. Here are the additional messages, in case helpful:

(EQ) PS C:\Users\Codiga_D> ncdump -h http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc
*   Trying 140.172.38.12:80...
* Connected to psl.noaa.gov (140.172.38.12) port 80 (#0)
> GET /thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc.dds HTTP/1.1
Host: psl.noaa.gov
User-Agent: oc4.8.1
Accept: */*

* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.23.1
< Date: Fri, 05 Aug 2022 19:40:36 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive
< Location: https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc.dds
< X-Frame-Options: SAMEORIGIN
<
* Ignoring the response-body
* Connection #0 to host psl.noaa.gov left intact
* Clear auth, redirects to port from 80 to 443
* Issue another request to this URL: 'https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc.dds'
*   Trying 140.172.38.12:443...
* Connected to psl.noaa.gov (140.172.38.12) port 443 (#1)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* schannel: added 142 certificate(s) from CA file 'C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\Lib\site-packages\certifi\cacert.pem'
* schannel: CertGetCertificateChain trust error CERT_TRUST_REVOCATION_STATUS_UNKNOWN
* Closing connection 1
Error:curl error: SSL peer certificate or SSH remote key was not OK
curl error details:
Warning:oc_open: Could not read url
C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\Library\bin\ncdump.exe: http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc: NetCDF: I/O failure

[DennisHeimbigner]

Error:curl error: SSL peer certificate or SSH remote key was not OK

This looks suspicious to me. Perhaps, the key in your known-hosts file for this site is out-of-date?

[DennisHeimbigner]

If you put this url in your browser, doe it succeed?

https://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc.dds

[ghost]

What it gives me is shown below. I believe that is success?

[DennisHeimbigner]

Yes, that shows that the cert chain used by your browser is working for reading from that site. Just to be sure, retry with HTTP.SSL.CAINFO and/or HTTP.SSL.CAPATH removed from .ncrc.

[ghost]

Okay, I removed the HTTP.SSL.CAINFO from .ncrc file and reran ncdump, the result is same as shown above.

[DennisHeimbigner]

I guess I am out of ideas. BTW, the above says that certs are being added from

• schannel: added 142 certificate(s) from CA file 'C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\Lib\site-packages\certifi\cacert.pem'

Any idea where those are coming from?

[ghost]

That file contains the certs that are used by various other tools including the python package "requests", which is working fine.

[ghost]

So thank you for all the help. At this point is there a way to summarize what we do and don't know about the situation? For example: "cert chain to the target site is working successfully in browser, other python packages such as requests are able to function fine, but something about cert chain(s) used by ncdump and/or curl causes the error". Is that more or less accurate? (I have minimal proficiency with certification issues. Just want to be able to explain what's happening, when seeking further help.)

[DennisHeimbigner]

The problem is not ncdump or other stand-alone use of the netcdf-c library. They work fine for me. The bottom is that the cert chains being used when python invokes netcdf-c is wrong somehow.

[paapu88]

I confirm this problem with netCDF4==1.6.0 (Ubuntu 22.04)

After saying pip install netcdf4==1.5.7 problem disappeared, the resulting environment (that works fine) is:

pip freeze

certifi==2022.6.15
cftime==1.6.1
cycler==0.11.0
fonttools==4.37.1
kiwisolver==1.4.4
matplotlib==3.5.3
netCDF4==1.5.7
numpy==1.23.2
packaging==21.3
pandas==1.4.3
Pillow==9.2.0
pyparsing==3.0.9
pyproj==3.3.1
python-dateutil==2.8.2
pytz==2022.2.1
six==1.16.0
xarray==2022.6.0

Terveisin, Markus

[DennisHeimbigner]

After re-reading this chain, I note that I was wrong and that HTTP.SSL.CAINFO apparently does not take the same argument as HTTP.SSL.CAPATH. CAPATH points to the containing directory and CAINFO points to the actual cert file. So you might set up .ncrc with the correct HTTP.SSL.CAINFO and see what happens.

[ghost]

I tried some other permutations of CAPATH (path only) and CAINFO (filename only, or path with filename) but still no joy. I also tried HTTP.SSL.VERIFYPEER=0 and HTTP.SSL.VALIDATE=0 but no success. Thank you for circling back with the suggestion @DennisHeimbigner.

[DennisHeimbigner]

Sorry that we could not solve this. I am out of ideas.

[rmcd-mscb]

I came to this thread because I was having this same issue accessing daymet using xarray in a conda environment (xr.open_dataset('https://thredds.daac.ornl.gov/thredds/dodsC/daymet-v4-agg/na.ncml'). Many other OPeNDAP endpoints worked but not daymet.

I'm working on WSL2 Ubuntu 22.04 and getting the same curl and netcdf error as @DanCodigaMWRA was experiencing in his conda example above using both netcdf4 v 1.6.1 and 1.6.0. Finally - 1.5.8 worked.

However, using curl in my terminal outside of my conda env I was able to access the daymet endpoint. Still curious, I added my ssl cert to conda `conda config --set ssl_verify <path/to/cert.crt> and now I can access using netcdf v1.6.1.

adding my experience in case someone else bumps into this thread.