Mads R. B. Kristensen

Closed in favor of https://github.com/rapidsai/cudf/pull/11553

I think this is a great idea!

Try `./build.sh libkvikio`, this PR only changes the C++ header-only library.

> C++ support for Batch IO was done in PR ( #220 ), right? Or is this about Python support? Yes, updated the issue

> If the HDF5 calls are single-threaded, you can presumably do whatever you like internally as long as you expose a "single-thread consistent" interface to HDF5. Correct, the VFD itself...

> The more fundamental problem for Legate is that we would have multiple processes trying to read/write the same HDF5 file; can the VFD approach handle that mode? In principle,...

> So at this point I believe the question is, is it better to go through the "official" VFD extension interface, or only use the HDF5 API up to the...

The only side-effect of `get_global_manager()`, when spilling is disabled, is the import of `cudf`. Sure enough, this also triggers the issue: ```diff diff --git a/dask_cuda/is_spillable_object.py b/dask_cuda/is_spillable_object.py index cb85248..959d9f3 100644 ---...

> I'm thinking instantiating `ProxifyHostFile` should be a fixture that is only set for tests that actually need it. Are there any reasons why we shouldn't be doing that? I...

> Would this be a "custom" extension? I.e. a third-party app that receives a kvikio-compressed HDF5 file wouldn't know what the attributes mean, therefore it wouldn't know that the file...