HDF5.jl
HDF5.jl copied to clipboard
JuliaIO
Should HDF5.jl support writing arrays of Float16 type?
Right now it doesn't work because Float16 is not part of the BitsKindOrString
It would be fine to support it as long as it can be done unambiguously (for example, in a way that "just works" if you import them in another programming language). If not, then better to use JLD.
JLD supports them just fine, but doesn't allow to mmap an array of Float16s, because it is an HDF5Opaque type. This was my original motivation for asking for support for Float16.
Do you think relaxing mmap constraints is a better solution? This essentially boils down to Float16 not being a BitsKindOrString as well though.
Oh, gotcha. I'm definitely fine with adding mmapable support to JLD, and to HDF5 if there's some agreed-upon standard for how Float16 should be written.
I should clarify that I won't be implementing this myself, and that it would make a great contribution.
I guess as long as Julia has consistent format for writing Float16s (which it seems it has), then doing for JLD (which is Julia-specific) shouldn't be a problem. I worked around it by reinterpreting it as UInt16, saving, mmaping, and reinterpreting back and it seems to work fine.
Any idea what would it take to support it natively? It seems like special-casing it and going through reinterpret should be doable entirely in JLD, but that sounds a bit ugly...
Nice!
To automate this, you could use readas and writeas, see https://github.com/JuliaIO/JLD.jl/blob/master/doc/jld.md#custom-serialization.
HDF5 allows defining arbitrary floating-point types; see https://www.hdfgroup.org/HDF5/doc/H5.user/Datatypes.html. To write them, one needs to create a new HDF5 atomic datatype (that's a few function calls) specifying how many exponent and mantissa bits there are in a Float16. When reading, these floats will automatically be converted to whatever floating-point representation the reader uses -- either the same Float16, or a different Float16 (if one of them is not IEEE compliant), or maybe Float32 or Float64 etc.
It would similarly be easy to support Int128 and UInt128.
I would be interested in contributing to make Float16 support happen. Has anyone seen an existing attempt or any fork / branch you have seen that I should look at before diving into this ?
