HDF5 include path detection issue

[thehesiod]

by default setup.py will use nc-config to detect all the include paths to use when cythonizing the pyx file. This is not completely correct because nc-config will only include headers for linking against libnetCDF, and not HDF5. Given that HDF5 is included directly (H5public.h), the include for this header should always be searched for separately. On debian for example this is in: /usr/include/hdf5/serial.

[jswhit]

We only need hdf5 headers to get the version number - maybe a better solution is to figure out how to get the version number from a library call. nc-config does detect the hdf5 library path.

[jswhit]

looks like we could use this H5get_libversion

https://support.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-Version

and get rid of the H5public.h include in netCDF4.pxi.

[jswhit]

Never mind - we still would need to access the hdf5 headers when compiling if we got the version number with a library call. However, looking at the nc-config utility, it seems that the --cflags option returns an include path that includes both the netcdf-4 install path and the CPPFLAGS used when building netcdf-c (which should include the path to the HDF5 headers). Too bad this doesn't work on Debian.

[thehesiod]

I think the issue is that when building netcdf I built specifying the path to the HDF5 header, and not a cflag specifying a new include path. If it works for you we can just close this...perhaps worth a comment that you need to build netcdf w/ HDF5 a certain way.

[thehesiod]

I was building it this way, and it resulted w/o settings in nc-config:

cmake .. -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=${INSTALL_PREFIX} -DHAVE_HDF5_H=/usr/include/hdf5/serial/hdf5.h -DMFHDF_H_INCLUDE_DIR=/usr/include/hdf -DENABLE_HDF4=ON

[jswhit]

Yep - nc-config only adds CPPFLAGS when you build with ./configure;make;make install, not when you use cmake. Sigh.

The problem with using the header to get the library version is that it is easy to pick up a header that is different from the one used to actually compile the library.

I've created a new pull request (pull request #679) that adds your header search, but uses the H5get_libversion call to get the version number to avoid this problem.

[thehesiod]

btw I noticed the HDF5_* environment variables are not used if nc-config/pkg_config are used which is kinda worrisome. I'd imagine it should assert if they're specified and instead nc-config/pkg-config are used, or they always take precedence.

[jswhit]

Maybe we should remove the env vars and just rely on setup.cfg.

[deeplycloudy]

I'm using HDF5 (and netcdf4) as built and installed via Anaconda, and with #679 the build process no longer automatically detects the HDF5 libraries. This step was not necessary for me to build from source in the past.

I was able to work around the issue by adding HDF5_dir = /Users/ebruning/anaconda/envs/ncblank to setup.cfg. If this is the desired approach, does the manual config step merit a mention in the Quick Start section of the README?

Here is the output before I added the HDF5 path.

# packages in environment at /Users/ebruning/anaconda/envs/ncblank:
#
curl                      7.49.0                        1  
cython                    0.25.2                   py36_0  
h5py                      2.7.0               np113py36_0  
hdf4                      4.2.12                        1  
hdf5                      1.8.17                        1  
jpeg                      9b                            0  
libnetcdf                 4.4.1                         1  
mkl                       2017.0.1                      0  
netcdf4                   1.2.4               np113py36_1  
numpy                     1.13.0                   py36_0  
openssl                   1.0.2l                        0  
pip                       9.0.1                    py36_1  
python                    3.6.1                         2  
readline                  6.2                           2  
setuptools                27.2.0                   py36_0  
six                       1.10.0                   py36_0  
sqlite                    3.13.0                        0  
tk                        8.5.18                        0  
wheel                     0.29.0                   py36_0  
xz                        5.2.2                         1  
zlib                      1.2.8                         3 

(ncblank) kenwood:netcdf4-python ebruning$ python setup.py install
reading from setup.cfg...
/Users/ebruning/anaconda/envs/ncblank/bin/nc-config: line 197: pkg-config: command not found
using nc-config ...
/Users/ebruning/anaconda/envs/ncblank/bin/nc-config: line 197: pkg-config: command not found

    HDF5_DIR environment variable not set, checking some standard locations ..
checking /Users/ebruning ...
checking /usr/local ...
checking /sw ...
checking /opt ...
checking /opt/local ...
checking /usr ...
Traceback (most recent call last):
  File "setup.py", line 341, in <module>
    _populate_hdf5_info(dirstosearch, inc_dirs, libs, lib_dirs)
  File "setup.py", line 306, in _populate_hdf5_info
    raise ValueError('did not find HDF5 headers')
ValueError: did not find HDF5 headers