How to handle CFP API across versions and install modalities

[markcmiller86]

ZFP can be built using either CMake or apparently also vanilla Make (GNU make).

But, these two systems are not consistent in how they ultimately install a ZFP build.

There is no install target for GNU make whereas there is for CMake. In addition, for CMake one can choose the install point. The contents of include and lib dirs from a mkdir build; cd build; cmake ..; make install differ significantly from those using just vanilla GNU make.

Its hard to anticipate how users will install ZFP and then where header and lib files can be found in the install point.

So, which of these is supported?

[lindstro]

GNU make builds are unfortunately very limited in this respect. The main use case for such builds is the ability to just type make and quickly obtain a static C library (lib/libzfp.a) and executable (bin/zfp) without the additional steps needed by CMake. Furthermore, GNU make does not support zfp components such as CUDA/HIP execution, Python bindings, and the 4300+ tests available on the develop branch--you need CMake for this.

I would welcome any contributions to extend the GNU make capabilities to also support installation of the subset of zfp that it supports. With the recent reorganization of directory structure in zfp 1.0.0, this should amount to little more than copying bin/, include/, and lib/ to where you want zfp installed. Perhaps something needs to be done about the Fortran module that ends up in modules/. In what respect does this differ significantly from the components installed by CMake?

[markcmiller86]

Maybe things aren't as different as I was seeing when looking at 0.5.4 and 0.5.5. Below I list contents of include, lib, and bin dirs of the two types of builds (and installs) for a few versions.

It looks like 1.0.0 is pretty close to same for both cases though now, at least for vanilla builds.

I am mainly concerned about a) ensuring H5Z-ZFP continues to build and operate correctly with zfp a few versions back and b) and know how to include and where to find header files and that I don't need different logic depending on how someone choose to install ZFP.

ZFP 0.5.4 - GNU Make with CFP_BUILD = 1 in Config

[scratlantis:~/silo/zfp-0.5.4] miller86% ls -R ./{include,lib,bin}
./bin:
testzfp	zfp

./include:
bitstream.h	zfp		zfp.h

./include/zfp:
macros.h	system.h	types.h

./lib:
libcfp.a	libzfp.a

ZFP-0.5.4 - CMake with -DBUILD_CFP:BOOL=ON

[scratlantis:~/silo/zfp-0.5.4] miller86% ls -R build/myinstall/{include,lib,bin}
ls: build/myinstall/bin: No such file or directory
build/myinstall/include:
bitstream.h	zfp		zfp.h		zfparray.h	zfparray1.h	zfparray2.h	zfparray3.h	zfpcodec.h	zfpcodecd.h	zfpcodecf.h

build/myinstall/include/zfp:
cache.h		iterator2.h	macros.h	pointer1.h	pointer3.h	reference2.h	system.h	view1.h		view3.h
iterator1.h	iterator3.h	memory.h	pointer2.h	reference1.h	reference3.h	types.h		view2.h

build/myinstall/lib:
cmake			libzfp.0.5.4.dylib	libzfp.0.dylib		libzfp.dylib

ZFP-0.5.5 - GNU Make with CFP_BUILD = 1 in Config

[scratlantis:~/silo/zfp-0.5.5] miller86% ls -R ./{include,lib,bin}
./bin:
testzfp	zfp

./include:
bitstream.h	zfp		zfp.h

./include/zfp:
macros.h	system.h	types.h

./lib:
libcfp.a	libzfp.a

ZFP-0.5.5 - CMake with -DBUILD_CFP:BOOL=ON

[scratlantis:~/silo/zfp-0.5.5] miller86% ls -R build/myinstall/{include,lib,bin}
ls: build/myinstall/bin: No such file or directory
build/myinstall/include:
bitstream.h	cfparray1f.h	cfparray2f.h	cfparray3f.h	zfp		zfparray.h	zfparray2.h	zfpcodec.h	zfpcodecf.h
cfparray1d.h	cfparray2d.h	cfparray3d.h	cfparrays.h	zfp.h		zfparray1.h	zfparray3.h	zfpcodecd.h	zfpfactory.h

build/myinstall/include/zfp:
cache.h		headerHelpers.h	iterator2.h	macros.h	pointer1.h	pointer3.h	reference2.h	system.h	view1.h		view3.h
header.h	iterator1.h	iterator3.h	memory.h	pointer2.h	reference1.h	reference3.h	types.h		view2.h

build/myinstall/lib:
cmake			libcfp.0.dylib		libzfp.0.5.5.dylib	libzfp.dylib
libcfp.0.5.5.dylib	libcfp.dylib		libzfp.0.dylib

ZFP-1.0.0 - GNU Make with CFP_BUILD = 1 in Config

[scratlantis:~/silo/zfp-1.0.0] miller86% ls -R ./{include,lib,bin}
./bin:
testzfp	zfp

./include:
zfp	zfp.h	zfp.hpp

./include/zfp:
array.h		array1.hpp	array3.hpp	bitstream.h	codec		constarray2.hpp	constarray4.hpp	index.hpp	version.h
array.hpp	array2.hpp	array4.hpp	bitstream.inl	constarray1.hpp	constarray3.hpp	factory.hpp	internal

./include/zfp/codec:
gencodec.hpp	zfpcodec.hpp

./include/zfp/internal:
array	cfp	codec	zfp

./include/zfp/internal/array:
cache.hpp	cache4.hpp	handle3.hpp	iterator2.hpp	pointer1.hpp	reference1.hpp	store.hpp	store4.hpp	view3.hpp
cache1.hpp	exception.hpp	handle4.hpp	iterator3.hpp	pointer2.hpp	reference2.hpp	store1.hpp	traits.hpp	view4.hpp
cache2.hpp	handle1.hpp	header.hpp	iterator4.hpp	pointer3.hpp	reference3.hpp	store2.hpp	view1.hpp
cache3.hpp	handle2.hpp	iterator1.hpp	memory.hpp	pointer4.hpp	reference4.hpp	store3.hpp	view2.hpp

./include/zfp/internal/cfp:
array1d.h	array1f.h	array2d.h	array2f.h	array3d.h	array3f.h	array4d.h	array4f.h	header.h

./include/zfp/internal/codec:
genheader.hpp	zfpheader.hpp

./include/zfp/internal/zfp:
inline.h	macros.h	system.h	types.h

./lib:
libcfp.a	libzfp.a

ZFP-1.0.0 - CMake with -DBUILD_CFP:BOOL=ON

[scratlantis:~/silo/zfp-1.0.0] miller86% ls -R build/myinstall/{include,lib,bin}
build/myinstall/bin:
zfp

build/myinstall/include:
zfp	zfp.h	zfp.hpp

build/myinstall/include/zfp:
array.h		array1.hpp	array3.hpp	bitstream.h	codec		constarray2.hpp	constarray4.hpp	index.hpp	version.h
array.hpp	array2.hpp	array4.hpp	bitstream.inl	constarray1.hpp	constarray3.hpp	factory.hpp	internal

build/myinstall/include/zfp/codec:
gencodec.hpp	zfpcodec.hpp

build/myinstall/include/zfp/internal:
array	cfp	codec	zfp

build/myinstall/include/zfp/internal/array:
cache.hpp	cache4.hpp	handle3.hpp	iterator2.hpp	pointer1.hpp	reference1.hpp	store.hpp	store4.hpp	view3.hpp
cache1.hpp	exception.hpp	handle4.hpp	iterator3.hpp	pointer2.hpp	reference2.hpp	store1.hpp	traits.hpp	view4.hpp
cache2.hpp	handle1.hpp	header.hpp	iterator4.hpp	pointer3.hpp	reference3.hpp	store2.hpp	view1.hpp
cache3.hpp	handle2.hpp	iterator1.hpp	memory.hpp	pointer4.hpp	reference4.hpp	store3.hpp	view2.hpp

build/myinstall/include/zfp/internal/cfp:
array1d.h	array1f.h	array2d.h	array2f.h	array3d.h	array3f.h	array4d.h	array4f.h	header.h

build/myinstall/include/zfp/internal/codec:
genheader.hpp	zfpheader.hpp

build/myinstall/include/zfp/internal/zfp:
inline.h	macros.h	system.h	types.h

build/myinstall/lib:
cmake			libcfp.1.dylib		libzfp.1.0.0.dylib	libzfp.dylib
libcfp.1.0.0.dylib	libcfp.dylib		libzfp.1.dylib

[lindstro]

It looks like 1.0.0 is pretty close to same for both cases though now, at least for vanilla builds.

Right. The main difference is that CMake creates shared objects by default whereas the GNU make builds static libraries. You can change this using make BUILD_SHARED_LIBS=1. However, there appears to be an issue with building libcfp in shared mode. I will look into this. Also, you don't get the symlinks that you do with CMake, and you don't get platform dependent extensions (e.g., .dylib and not .so on Mac). But we could probably quite easily add the symlinks by parsing zfp/version.h, which would be appropriate for an install target. Let me take a look at how you're doing this in H5Z-ZFP.

I am mainly concerned about a) ensuring H5Z-ZFP continues to build and operate correctly with zfp a few versions back and b) and know how to include and where to find header files and that I don't need different logic depending on how someone choose to install ZFP.

I think you should be fine if you just include zfp.h. From there you can examine the ZFP_VERSION_* macros for any version-specific behavior. To my knowledge, nothing has changed with respect to what goes in bin/ and lib/ over the past several versions (0.5.x to 1.0.0), as your directory listings also seem to confirm.

[markcmiller86]

I think you should be fine if you just include zfp.h. From there you can examine the ZFP_VERSION_* macros for any version-specific behavior.

I don't think that is true for CFP api is it? The headers have changed name and location I think. In 0.5.4 and 0.5.5 with GNU Make build, they are not copied to top-level include dir and so including them from that kind of install involves different logic than including them from a CMake install. In 1.0.0, they appear to be handled same in both kinds of installs.

I guess that is the real crux of my issue here...how to handle CFP API across those versions and install modalities. That said, I now don't think its too critical either. I mean, my testing logic is more convoluted but I could opt to not worry about testing H5Z-ZFP on earlier versions of ZFP.

[lindstro]

I know we have discussed this in the past, but please remind me: Why do you need cfp? Looking at the H5Z-ZFP repo, I see it used only in a test. Is it being used to generate zfp-compressed data, or is it testing the ability of H5Z-ZFP to serialize compressed-array classes?

It looks like you made some changes to handle the refactored cfp that was on develop before the final 1.0.0 release, which includes some additional changes, including the renaming of include/cfp.h to include/zfp/array.h. We did this to have a corresponding C and C++ file for similar functionality, e.g., zfp.h and zfp.hpp, zfp/array.h and zfp/array.hpp.

Now, to handle this properly, I would simply include zfp.h and then use #ifdefs as appropriate to handle different versions. For example,

#include "zfp.h"
#if ZFP_VERSION_MAJOR < 1
  #include "cfp.h"
#else
  #include "zfp/array.h"
#endif

[markcmiller86]

Why do you need cfp?

To test H5Z-ZFP's ability to read (and write) compressed arrays written to HDF5 without going through the normal HDF5 filtering logic (which would require that what is handed to HDF5 is NOT already compressed). Currently, I have only a test for the write case (and readback that it compares correctly). I also want to test the read case (e.g. reading into memory an array without UNcompressing it).

So, H5Z-ZFP doesn't need it really at all. But, I do want to test H5Z-ZFP in the presence of CFP arrays so make sure we can handle them.

[lindstro]

You can change this using make BUILD_SHARED_LIBS=1. However, there appears to be an issue with building libcfp in shared mode. I will look into this.

As it turns out, I forgot that we already addressed this, but the fix is not well documented. The issue seems limited to macOS, and the fix is to add OS=mac. I filed a new issue (#170) to ensure proper documentation is added.

[lindstro]

@markcmiller86 Can we close this issue or are there other things you'd like to see addressed? Regarding GNU make installation, I've filed a separate issue, #172. The subject line of this issue refers to CFP API changes, which I believe can be dealt with using zfp's VERSION macros, as outlined above.