hdf5-rust

hdf5-rust copied to clipboard

Since you have the file interface already implemented, it's very easy to add reading files from buffer. For some applications, it accelerates the process of I/O a lot.

All you have to do is, instead of opening a file like this:

  fileHandle = H5Fopen(filename, H5F_ACC_RDONLY, H5P_DEFAULT);

Allow it with this (in C++ terms):

fileHandle = H5LTopen_file_image((void *)&FileImage.front(), FileImage.size(), H5LT_FILE_IMAGE_OPEN_RW);

In my C++ code I have an abstraction layer that does this. Everything else is the same. Of course, this assumes the file will not be modified, because modifying an image requires implementing a C interface that would realloc and do other things. I haven't done that part before myself because I didn't need it. But for now, it can be ignored, because most people who need to access files from memory only need reading.

H5LT* functions are not part of libhdf5, they're part of the "high-level HDF5 library", i.e. libhdf5_hl. We haven't touched that yet, nor has it been wrapped and linked to... I guess we could - but it will need to be done separately.

Looking at the source code though... it uses core driver + H5Pset_file_image_callbacks() (which, IIRC, is the only FAPL property that hasn't been exposed here, there's very low probability someone would go on writing their own file drivers and all the callbacks etc).

For some applications, it accelerates the process of I/O a lot.

If you don't need direct access to the image, you can use the core driver directly without it being filebacked? This doesn't require libhdf5_hl.

Otherwise, someone will need to generate bindings for (various versions of) libhdf5_hl, check that it can be linked on all platforms, etc...

If you don't need direct access to the image, you can use the core driver directly without it being filebacked? This doesn't require libhdf5_hl.

What do you mean with direct access? Can the core driver do the same with an image, like read it directly from a buffer?

Btw, for my usage, I never set any callbacks. That's just for read-only. But yes, I used the hl library too.

Core driver is ok if you just want to create something in memory and work with it, IIUC, but it doesn't immediately let you read/write to in-memory images, for that you need to set callbacks (which H5Pset_file_image_callbacks does and which is what H5LTopen_file_image calls)...

But anyways - yea, if/when we provide bindings for libhdf5_hl, we could easily support that. I'm not sure though, since it's a separate library, it may exist or it may not exist, so the build process will become slightly more complicated.