c-blosc
c-blosc copied to clipboard
Blosc
The HDF5 filter fails tests on Windows when c-blosc and HDF5 link to different CRTs
When you build c-blosc on Windows, the HDF5 filter test will fail if c-blosc and HDF5 link to different versions of the C run-time library (CRT). The reason for this is that, unlike systems based on Unix, Windows does not make the C library (C run-time) a part of the operating system. Instead, the C library is implemented in separate shared libraries (dlls - e.g. msvcr110.dll). Each version of Visual Studio uses a different set of dlls and they are further split into debug and release versions. None of these dlls share CRT state (heap management structures, etc.) Problems thus arise when memory is allocated in one dll and freed in another.
This problem is inherent to how the H5Z functions in HDF5 handle memory and is not a c-blosc issue. The HDF Group will have to work out a solution for this, which will possibly involve exposing H5Z re/c/m/alloc and free functions and ensuring that buffers are correctly handled in the H5Z code.
In the meantime, users should be aware that the binaries released by THG are release only, so running the HDF5 filter tests when c-blosc is built in debug mode will always fail when linking against those. They do pass when both c-blosc and HDF5 are built with the same configuration (release vs. debug).
OMG, this is pretty bad. Hopefully you can solve the release vs debug issue on Windows soon. Meanwhile perhaps documenting this in the hdf5/README file would be in order. Do you want to do that?
2014-11-18 5:20 GMT+01:00 Dana Robinson [email protected]:
When you build c-blosc on Windows, the HDF5 filter test will fail if c-blosc and HDF5 link to different versions of the C run-time library (CRT). The reason for this is that, unlike systems based on Unix, Windows does not make the C library (C run-time) a part of the operating system. Instead, the C library is implemented in separate shared libraries (dlls - e.g. msvcr110.dll). Each version of Visual Studio uses a different set of dlls and they are further split into debug and release versions. None of these dlls share CRT state (heap management structures, etc.) Problems thus arise when memory is allocated in one dll and freed in another.
This problem is inherent to how the H5Z functions in HDF5 handle memory and is not a c-blosc issue. The HDF Group will have to work out a solution for this, which will possibly involve exposing H5Z re/c/m/alloc and free functions and ensuring that buffers are correctly handled in the H5Z code.
In the meantime, users should be aware that the binaries released by THG are release only, so running the HDF5 filter tests when c-blosc is built in debug mode will always fail when linking against those. They do pass when both c-blosc and HDF5 are built with the same configuration (release vs. debug).
— Reply to this email directly or view it on GitHub https://github.com/Blosc/c-blosc/issues/70.
Francesc Alted
I'll edit the README and I've already raised a flag at The HDF Group. Hopefully we can get a fix out soon.
I just added some extra functions to HDF5 that expose the library's malloc, etc. and tested it with c-blosc and it appears to work. I can now build c-blosc in release while using HDF5 built in debug, for example. It needs more testing, but it looks like this will solve the problem.
I'll see about getting the new functionality added to HDF5 1.8.15 (May 2015) but in the meantime I'll document the issue in your README.
Very good news. Thanks for doing that! Feel free to update the hdf5/README.rst and I will merge your PR.
I added a detailed description to the README in the HDF5 filter directory. It's all in the pull request, which I have updated.
Ok, so let's keep this issue open and will close it as soon as HDF5 1.8.15 would be out. Thanks.
FWIW, we'll have this fixed in Anaconda, because we will be building HDF5 and blosc ourselves, and keeping MSVC versions consistent with the standards for Python (2008 for py 2.7, and so on)
I just came across this issue while browsing around for something else.
In my case, I'm actually using the same VS version (2015) to build Blosc and the Blosc HDF5 plugin as the one that was used by The HDF5 Group to build the pre-built binaries (1.8.18), so I guess there's currently no risk of me running into problems.
But anyone know what the status is, was the memory management functions exposed upstream and the Blosc plugin updated? Or could I expect trouble if I a) move to another VS version, or the HDF5 Group does, and b) I use release + debug of Blosc / HDF5 (or the other way around) ?
