Authentication for OpenDAP

[davidorme]

Hi,

I'm using netcdf4-python (1.5.1.2) on a Mac (OS 10.14.4), with netcdf4 installed via brew with dap2 support:

$ nc-config --has-dap2 --version
yes
netCDF 4.6.2

I'm trying to access files via OpenDAP on CEDA (https://data.ceda.ac.uk), which require registration. I've used their instructions to generate a certificate:

https://help.ceda.ac.uk/article/4442-ceda-opendap-scripted-interactions

I have then set up a .dodsrc file pointing to the certificate and the trust roots.

HTTP.COOKIEJAR=./dods_cookies
HTTP.SSL.CERTIFICATE=./credentials.pem
HTTP.SSL.KEY=./credentials.pem
HTTP.SSL.CAPATH=./ca-trustroots

If I now try and access a file from within python then I get an I/O error:

In [1]: import netCDF4
   ...: ds = netCDF4.Dataset('http://dap.ceda.ac.uk/thredds/dodsC/badc/cmip5/data/cmip5/output1/ICHEC/EC-EARTH/rcp85/day/atmos/day/r12i1p1/latest/tas/tas_day_EC-EARTH_rcp85_r12i1p1_20560101-21001231.nc')
   ...: 
   ...: ds.variables.keys()
   ...: 
curl error details: 
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-1-1de901bc5de0> in <module>()
      1 import netCDF4
----> 2 ds = netCDF4.Dataset('http://dap.ceda.ac.uk/thredds/dodsC/badc/cmip5/data/cmip5/output1/ICHEC/EC-EARTH/rcp85/day/atmos/day/r12i1p1/latest/tas/tas_day_EC-EARTH_rcp85_r12i1p1_20560101-21001231.nc')
      3 
      4 ds.variables.keys()
netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Dataset.__init__()
netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success()
IOError: [Errno -68] NetCDF: I/O failure: 'http://dap.ceda.ac.uk/thredds/dodsC/badc/cmip5/data/cmip5/output1/ICHEC/EC-EARTH/rcp85/day/atmos/day/r12i1p1/latest/tas/tas_day_EC-EARTH_rcp85_r12i1p1_20560101-21001231.nc'

However, if I run the same path with ncdump, then I get a successful connection, the data is downloaded and the dods_cookies file is populated with session tokens. After this, netCDF4-python works fine. That makes it seem like netCDF4-python knows about the .dodsrc file and can use the session cookies that it points to but can't currently generate those cookies. Or alternatively, there is a setup step I'm missing in Python.

Cheers, David

[jswhit]

I'm pretty sure that netcdf4-python goes through the same code path in the C library as the ncdump utility does. Are you sure that netcdf4-python is built with the same version of netcdf-c as the ncdump you are using?

@DennisHeimbigner, do you have any thoughts on this?

[rsmith013]

I have been having the same problem. netcdf4 installed with conda install -c anaconda netcdf4 uses version 13 of the C library.

When installed with pip, the latest version 1.5.1.2 uses v 16 of the C lib.

I have come up against this issue when using the pip installed netcdf library but it seems to work as expected when installed with conda.

[davidorme]

OK, so netcdf4-python is using its own version of netcdf4-c:

>>> import netCDF4
>>> netCDF4.__netcdf4libversion__
u'4.6.3'

If I look at the open files used by the Python process, it seems to be using the package version at /usr/local/lib/python2.7/site-packages/netCDF4/.dylibs/libnetcdf.15.dylib. I only know how to check the config using nc-config but I assume that the pip binaries support OpenDAP because libcurl is also in that folder and being linked by Python.

I could install from source so it uses my brew installed libs, but it seems like it should still work even if there are two different versions of the netcdf-c library?

[jswhit]

Could be that the netcdf-c library is configured and built differently in the Ubuntu package used by travis to build the binary wheels used by pip. Maybe the curl/openssl support is disabled, or different.

Or, is it actually using the netcdf library installed via homebrew? Maybe there is an issue with the openssl support in the homebrew curl library?

[davidorme]

I can rule out (I think!) some of that:

1. If I use ncdump (which is installed by brew) then I can access the CEDA OpenDAP files: ncdump successfully uses .dodsrc and the tags in it to negotiate session tokens and then fetches data. So I think the brew installation is working as expected.

2. I don't think curl/openssl can be disabled in the pip wheels because once the dods_cookies file exists via ncdump, then netCDF4-python accesses the files perfectly happily. I did wonder if it might be a path issue - my .dodsrc file and the certs are in a local folder not $HOME, but again, once the dods_cookies exist then netCDF4-python uses them. I guess that leaves different.

[jswhit]

You can use 'otool -L' on the ncdump executable and the _netCDF4.so shared object to see if they are using different versions of the curl and/or netcdf libs.

[davidorme]

Which gives:

$ otool -L /usr/local/bin/ncdump
/usr/local/bin/ncdump:
	@rpath/libnetcdf.13.dylib (compatibility version 13.0.0, current version 13.0.0)
	/usr/local/opt/hdf5/lib/libhdf5_hl.100.dylib (compatibility version 102.0.0, current version 102.1.0)
	/usr/local/opt/hdf5/lib/libhdf5.103.dylib (compatibility version 104.0.0, current version 104.0.0)
	/usr/local/opt/szip/lib/libsz.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)

and

$ otool -L /usr/local/lib/python2.7/site-packages/netCDF4/_netCDF4.so 
/usr/local/lib/python2.7/site-packages/netCDF4/_netCDF4.so:
	@loader_path/.dylibs/libnetcdf.15.dylib (compatibility version 16.0.0, current version 16.0.0)
	@loader_path/.dylibs/libhdf5_hl.100.dylib (compatibility version 102.0.0, current version 102.0.0)
	@loader_path/.dylibs/libhdf5.101.dylib (compatibility version 103.0.0, current version 103.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)

To me, that looks like the netCDF4-python binary wheel isn't using libcurl, although it is present in the package .dylibs directory, but that doesn't seem to square with netCDF4-python working when the cookies are present?

$ ls -lah  /usr/local/lib/python2.7/site-packages/netCDF4/.dylibs
total 35088
drwxr-xr-x  8 dorme  admin   256B 21 Jun 15:36 .
drwxr-xr-x  8 dorme  admin   256B 21 Jun 15:36 ..
-rw-r--r--  1 dorme  admin    75K 21 Jun 15:36 libaec.0.dylib
-rw-r--r--  1 dorme  admin   688K 21 Jun 15:36 libcurl.4.dylib
-rw-r--r--  1 dorme  admin    13M 21 Jun 15:36 libhdf5.101.dylib
-rw-r--r--  1 dorme  admin   358K 21 Jun 15:36 libhdf5_hl.100.dylib
-rw-r--r--  1 dorme  admin   3.0M 21 Jun 15:36 libnetcdf.15.dylib
-rw-r--r--  1 dorme  admin    33K 21 Jun 15:36 libsz.2.dylib

[jswhit]

How about 'otool -L /usr/local/lib/python2.7/site-packages/netCDF4/.dylibs/libnetcdf.15.dylib'?

[DennisHeimbigner]

You might re-review https://github.com/Unidata/netcdf-c/blob/master/docs/auth.md#REDIR to ensure that the python tests are following all the steps, especially for redirection. This note from that document should be verified:

One final note. In using this, you MUST to specify a real file in the file system to act as the cookie jar file (HTTP.COOKIEJAR) so that the redirect site can properly pass back authorization information.

[davidorme]

Oh, okay. _netCDF4.so links libnetcdf.15.dylib and that links libcurl.

$ otool -L /usr/local/lib/python2.7/site-packages/netCDF4/.dylibs/libnetcdf.15.dylib 
/usr/local/lib/python2.7/site-packages/netCDF4/.dylibs/libnetcdf.15.dylib:
	/DLC/netCDF4/libnetcdf.15.dylib (compatibility version 16.0.0, current version 16.0.0)
	@loader_path/libhdf5_hl.100.dylib (compatibility version 102.0.0, current version 102.0.0)
	@loader_path/libhdf5.101.dylib (compatibility version 103.0.0, current version 103.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
	@loader_path/libsz.2.dylib (compatibility version 3.0.0, current version 3.1.0)
	@loader_path/libaec.0.dylib (compatibility version 1.0.0, current version 1.4.0)
	@loader_path/libcurl.4.dylib (compatibility version 9.0.0, current version 9.0.0)
	/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1451.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.41.2)

[jswhit]

That's what I thought. The next question is does libcurl link openssl? It looks like maybe it uses the MacOSX security framework instead.

[davidorme]

Looks like it:

$ otool -L /usr/local/lib/python2.7/site-packages/netCDF4/.dylibs/libcurl.4.dylib 
/usr/local/lib/python2.7/site-packages/netCDF4/.dylibs/libcurl.4.dylib:
	/DLC/netCDF4/libcurl.4.dylib (compatibility version 9.0.0, current version 9.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1451.0.0)
	/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.41.2)
	/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)

[davidorme]

Hi @DennisHeimbigner,

I've tried playing around with the dods_cookies file specified in .dodsrc. If it doesn't exist, then attempting an OpenDAP connection with netCDF4-python does create an empty file with that name, but does not / is not able to populate it with the session cookies and tokens.

[DennisHeimbigner]

If you pre-create dods_cookies with the correct permissions, I assume it works, correct?

[DennisHeimbigner]

I think we encountered this problem before where on OSX,where we needed to use the osx replacements for openssl. Since we (unidata) do not regularly test authorization, we might not have checked this any time recently. The curl website probably needs to be examined to see what it says about this problem.

[DennisHeimbigner]

BTW, when this is solved, It would help if someone provided a concise description of the fix so I can add it to our (netcdf-c) documentation to help others with this problem.

[davidorme]

@DennisHeimbigner: That's correct. Because I've got the parallel installation of netCDF4-c and the command line tools via brew, ncdump correctly creates the dods_cookies file and, once it is there, netCDF4-python uses it and works nicely.

[jswhit]

I bet the libcurl used by the netcdf-c library installed by homebrew uses a homebrew-built openssl, not the MacOSX security framework.

[davidorme]

That's not clear (to me). otool shows that ncdump links the /usr/local/lib/libnetcdf.dylib (as you'd expect). Like all (?) homebrew installations, that is a symlink to resources within /usr/local/Cellar:

lb-dormelap:Downloads dorme$ ls -lah /usr/local/lib/libnetcdf.dylib 
lrwxr-xr-x  1 dorme  admin    42B  4 Mar 16:23 /usr/local/lib/libnetcdf.dylib -> ../Cellar/netcdf/4.6.2/lib/libnetcdf.dylib

However, libnetcdf.dylib links /usr/lib/libcurl and then that links /usr/lib/libssl and neither of these are symlinked to /usr/local/Cellar. I know that the SIP protection on Mac includes /usr/lib and I didn't think homebrew used it, but I could be wrong.

$ otool -L /usr/local/lib/libnetcdf.dylib 
/usr/local/lib/libnetcdf.dylib:
	/usr/local/opt/netcdf/lib/libnetcdf.13.dylib (compatibility version 13.0.0, current version 13.0.0)
	/usr/local/opt/hdf5/lib/libhdf5_hl.100.dylib (compatibility version 102.0.0, current version 102.1.0)
	/usr/local/opt/hdf5/lib/libhdf5.103.dylib (compatibility version 104.0.0, current version 104.0.0)
	/usr/local/opt/szip/lib/libsz.2.dylib (compatibility version 3.0.0, current version 3.0.0)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
$ otool -L /usr/lib/libcurl.4.dylib 
/usr/lib/libcurl.4.dylib:
	/usr/lib/libcurl.4.dylib (compatibility version 7.0.0, current version 9.0.0)
	/usr/lib/libcrypto.42.dylib (compatibility version 43.0.0, current version 43.0.0)
	/usr/lib/libssl.44.dylib (compatibility version 45.0.0, current version 45.1.0)
	/System/Library/Frameworks/LDAP.framework/Versions/A/LDAP (compatibility version 1.0.0, current version 2.4.0)
	/System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos (compatibility version 5.0.0, current version 6.0.0)
	/usr/lib/libapple_nghttp2.dylib (compatibility version 1.0.0, current version 1.24.1)
	/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.250.1)

[jswhit]

@matthew-brett - the diagnosis here suggests that the libcurl build for macos x should not use --with-darwinssl