netcdf4-python
netcdf4-python copied to clipboard
Unidata
Failure to load OPeNDAP in Colab
Recently, reading OPeNDAP files when running notebooks on Google Colab resutss in an OSError: [Errno -68] NetCDF: I/O failure
Relevant info:
- netCDF4 version: 1.6.0
- Python version: Python 3.7.13
- Environment: Google Colab
Example code:
import netCDF4
ds = netCDF4.Dataset('http://thredds.socib.es/thredds/dodsC/hf_radar/hf_radar_ibiza-scb_codarssproc001/L1/2016/dep0001_hf-radar-ibiza_scb-codarssproc001_L1_2016-02.nc')
Error:
Colab Notebook link: https://colab.research.google.com/drive/17juZulxoz8sahoDJVuU4uOMSnBC_xXDA?usp=sharing
Additional information: When using "netCDF4<1.6.0", the code runs fine, without any errors.
I've run into this issue as well:
Additionally, it looks as though this issue only happens when in a pure python environment running netCDF4==1.6.0. My test ensemble based on Anaconda and netCDF4==1.6.0 seems to be opening this OPeNDAP URL perfectly fine!
Update: I think the wheel being supplied via PyPI is broken. After compiling netCDF4==1.6.0 from sources, everything seems to be working perfectly fine. I needed to run the following after other dependencies were installed in my environment.
$ python -m pip install --upgrade --force-reinstall --no-deps --no-cache-dir netcdf4 --no-binary netcdf4
You may want to consider rebuilding your wheels (for manylinux * x86_64, anyway).
For reference: https://github.com/CSHS-CWRA/RavenPy/pull/193/commits/5bab92e26f3abbae7e54651bf67685cdce89c208
You may want to consider rebuilding your wheels (for
manylinux*x86_64, anyway).
What specifically do you think needs to be done differently when building the manylinux wheels?
BTW, the screenshots in https://github.com/Unidata/netcdf4-python/issues/1179#issuecomment-1202720571 make it hard to read about the real issue. It seems that the problem is with curl and some SSL certificates. I get this when I try it locally:
Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details:
Warning:oc_open: Could not read url
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
@jswhit what version/build of curl are you using in the wheel?
It may be a version problem or something with the wheel build. I guess the only way to figure this out is to add an OPeNDAP test during the wheel test phase. This is how I reproduced it locally in case you want to check it out:
conda create --name TEST python=3 pip
pip install netcdf4
python -c "import netCDF4; ds = netCDF4.Dataset('http://thredds.socib.es/thredds/dodsC/hf_radar/hf_radar_ibiza-scb_codarssproc001/L1/2016/dep0001_hf-radar-ibiza_scb-codarssproc001_L1_2016-02.nc')"
Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details:
Warning:oc_open: Could not read url
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "src/netCDF4/_netCDF4.pyx", line 2353, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 1963, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -68] NetCDF: I/O failure: b'http://thredds.socib.es/thredds/dodsC/hf_radar/hf_radar_ibiza-scb_codarssproc001/L1/2016/dep0001_hf-radar-ibiza_scb-codarssproc001_L1_2016-02.nc'
Then, if I do a conda install netcdf4, the test pass and I have these packages installed:
Name Version Build Channel
──────────────────────────────────────────────────────────
curl 7.83.1 h2283fc2_0 conda-forge
hdf5 1.12.2 nompi_h4df4325_100 conda-forge
libcurl 7.83.1 h2283fc2_0 conda-forge
libnetcdf 4.8.1 nompi_h21705cb_103 conda-forge
netcdf4 1.6.0 nompi_py310h9fd08d4_101 conda-forge
openssl 3.0.5 h166bdaf_1 conda-forge
@ocefpaf @jswhit any way we could help here? FWIW, this issue is not present on M1 Macs.
I believe that the wheel should be rebuilt. One test folks could try is to conda install it on colab and see if that works.
I was having the same problem. I installed once with "%pip install netcdf4<1.6" and once with "%pip install netcdf4<1.6". I can confirm it works with one, but not the other.
For each, I used ldd to look at the linked libraries. Below are two lists. The first is the full list where I have sorted to align libraries. The second is just the places where version number or library has changed. I'm not sure if this helps, but I am hoping someone who knows more about this would have some thoughts.
--- v15.txt 2022-09-19 17:49:58.161994184 +0000
+++ v16.txt 2022-09-19 17:49:59.071994132 +0000
@@ -1,18 +1,20 @@
- linux-vdso.so.1 (0x00007ffecfb0d000)
+ linux-vdso.so.1 (0x00007ffdd1397000)
- /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 (0x00007f99aaa79000)
+ /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4 (0x00007fddc9e71000)
- libnetcdf-cb1b5117.so.18.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libnetcdf-cb1b5117.so.18.0.0 (0x00007f99aa8bf000)
+ libnetcdf-0ef85704.so.19 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libnetcdf-0ef85704.so.19 (0x00007fddc9b8c000)
- libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f99aa6a0000)
+ libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fddc996d000)
- libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f99aa2af000)
+ libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fddc957c000)
- libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007f99aa094000)
+ libunwind.so.8 => /usr/lib/x86_64-linux-gnu/libunwind.so.8 (0x00007fddc9361000)
- libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f99a9d0b000)
+ libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fddc8fd8000)
- libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f99a996d000)
+ libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fddc8c3a000)
- libhdf5_hl-f3ea9bc7.so.200.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5_hl-f3ea9bc7.so.200.0.0 (0x00007f99aad38000)
+ libhdf5_hl-f45ebdf9.so.200.1.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5_hl-f45ebdf9.so.200.1.0 (0x00007fddca12c000)
- libhdf5-3e8ef4e5.so.200.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5-3e8ef4e5.so.200.0.0 (0x00007f99a9382000)
+ libhdf5-115e850b.so.200.2.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5-115e850b.so.200.2.0 (0x00007fddc85cc000)
- libsz-57467d8a.so.2.0.1 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libsz-57467d8a.so.2.0.1 (0x00007f99aad31000)
+ libsz-57467d8a.so.2.0.1 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libsz-57467d8a.so.2.0.1 (0x00007fddca125000)
- libaec-e300f322.so.0.0.10 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libaec-e300f322.so.0.0.10 (0x00007f99aad25000)
+ libaec-e300f322.so.0.0.12 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libaec-e300f322.so.0.0.12 (0x00007fddca115000)
- libcurl-8c767087.so.4.7.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libcurl-8c767087.so.4.7.0 (0x00007f99a9095000)
+ libcurl-8c767087.so.4.7.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libcurl-8c767087.so.4.7.0 (0x00007fddc7e33000)
- libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f99a8e78000)
+ libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fddc83af000)
+ libzstd-c4cc7033.so.1.5.2 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libzstd-c4cc7033.so.1.5.2 (0x00007fddc8120000)
+ libblosc-a9e8a7c7.so.1.21.1 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libblosc-a9e8a7c7.so.1.21.1 (0x00007fddc8289000)
- libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f99a8c74000)
+ libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fddc8a36000)
- /lib64/ld-linux-x86-64.so.2 (0x00007f99aace9000)
+ /lib64/ld-linux-x86-64.so.2 (0x00007fddca0e1000)
- liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f99a8a4e000)
+ liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007fddc79f5000)
- libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f99a8836000)
+ libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fddc7c1b000)
Below is a more narrow list of just where versions have changed.
- libnetcdf-cb1b5117.so.18.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libnetcdf-cb1b5117.so.18.0.0 (0x00007f99aa8bf000)
+ libnetcdf-0ef85704.so.19 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libnetcdf-0ef85704.so.19 (0x00007fddc9b8c000)
- libhdf5_hl-f3ea9bc7.so.200.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5_hl-f3ea9bc7.so.200.0.0 (0x00007f99aad38000)
+ libhdf5_hl-f45ebdf9.so.200.1.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5_hl-f45ebdf9.so.200.1.0 (0x00007fddca12c000)
- libhdf5-3e8ef4e5.so.200.0.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5-3e8ef4e5.so.200.0.0 (0x00007f99a9382000)
+ libhdf5-115e850b.so.200.2.0 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libhdf5-115e850b.so.200.2.0 (0x00007fddc85cc000)
- libaec-e300f322.so.0.0.10 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libaec-e300f322.so.0.0.10 (0x00007f99aad25000)
+ libaec-e300f322.so.0.0.12 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libaec-e300f322.so.0.0.12 (0x00007fddca115000)
+ libzstd-c4cc7033.so.1.5.2 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libzstd-c4cc7033.so.1.5.2 (0x00007fddc8120000)
+ libblosc-a9e8a7c7.so.1.21.1 => /usr/local/lib/python3.7/dist-packages/netCDF4/../netCDF4.libs/libblosc-a9e8a7c7.so.1.21.1 (0x00007fddc8289000)
The newest release (1.6.1) hasn't fixed the problems for my use case either:
src/netCDF4/_netCDF4.pyx:2463: in netCDF4._netCDF4.Dataset.__init__
???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OSError: [Errno -68] NetCDF: I/O failure: b'https://pavics.ouranos.ca/twitcher/ows/proxy/thredds/dodsC/birdhouse/disk2/testdata/raven/raven-gr4j-cemaneige/Salmon-River-Near-Prince-George_meteo_daily.nc'
src/netCDF4/_netCDF4.pyx:2026: OSError
----------------------------- Captured stderr call -----------------------------
Error:curl error: Problem with the SSL CA cert (path? access rights?)
curl error details:
Warning:oc_open: Could not read url
@jswhit, I am available to do some digging and try to solve this issue since it is causing a lot of problem for other libraries. Do you have any suggestions to where should I start looking?
This is not a solution, just a work around, but you can try:
!pip install -q condacolab
import condacolab
condacolab.install()
import condacolab
condacolab.check()
!mamba install netcdf4
and then,
import netCDF4
ds = netCDF4.Dataset('http://thredds.socib.es/thredds/dodsC/hf_radar/hf_radar_ibiza-scb_codarssproc001/L1/2016/dep0001_hf-radar-ibiza_scb-codarssproc001_L1_2016-02.nc')
I'll check the openssl version used and try to add a regression test in the wheel building process to investigate. I can confirm that this doesn't happen in the Windows wheel and @ngam confirmed it doesn't happen in the M1 wheel. It must be the Linux openssl version then.
In the Aug 8 traceback above (and pasted below), the URL string gets turned into a byte string by netCDF4. Could that be relevant? I had this problem and fixed it by using an older version of netCDF4 that keeps the string as a string. There was a similar issue with strings converted to byte strings 2 years ago: https://github.com/pydata/xarray/issues/4859
Traceback (most recent call last):
File "
I just used the colab notebook pinned in the first comment and used the new netcdf4==1.6.4 version.
Everything is running as expected,
I think this issue was solved.
See https://github.com/Unidata/netcdf4-python/issues/1246#issuecomment-1577568553
There are a few overlapping issues that may be closed. Posting them here to facilitate closing later:
https://github.com/Unidata/netcdf4-python/issues/1199 https://github.com/Unidata/netcdf4-python/issues/812 https://github.com/Unidata/netcdf4-python/issues/753