H5TOOLS_BUFSIZE documentation error

[mkitti]

The documentation for h5repack states:

For example, if encountering a performance issue when using h5repack with a large 3D dataset with a chunk size of 512*512*512 and a datatype of 32-bit float (4 bytes in size), then setting H5TOOLS_BUFSIZE to the size of (at least) one chunk times the size of the datatype (512*512*512*4=536870912) should improve performance. On Unix the H5TOOLS_BUFSIZE environment variable can be set as follows:

setenv H5TOOLS_BUFSIZE 536870912

Furthermore, on a page about environmental variables it states:

Adjusting the h5repack buffer size If encountering poor performance using h5repack with large datasets, the H5TOOLS_BUFSIZE environment variable can be used to improve performance. This environment variable specifies the hyperslab (selection) buffer size (in bytes) that is used by h5repack.

https://portal.hdfgroup.org/display/HDF5/Environment+Variables+Used+by+HDF5

Both statements strong imply that H5TOOLS_BUFSIZE is the number of bytes of the buffer.

Looking at the source code, the function h5tools_getenv_update_hyperslab_bufsize reads H5TOOLS_BUFSIZE from the environment variable and then multiplies it by 1024 * 1024. It seems that H5TOOLS_BUFSIZE is really interpreted as the number of mebibytes.

https://github.com/HDFGroup/hdf5/blob/f875a73711ea125cbe67ea7d65facce669af96a4/tools/lib/h5tools_utils.c#L879-L889

Attempted to set H5TOOLS_BUFSIZE` to a large value in bytes would then result in an error as follows:

Error occurred while repacking
H5tools-DIAG: Error detected in HDF5:tools (1.12.2) thread 0:
  #000: C:\autotest\hdf5112-StdRelease-dist-10vs16\build\hdfsrc\tools\src\h5repack\h5repack_copy.c line 329 in copy_objects(): do_copy_objects from <c_2022_0427_113351_FNDR_X20Y12_rotation000_532 nm_vol001.raw> could not copy data to <chunked/c_2022_0427_113351_FNDR_X20Y12_rotation000_532 nm_vol001.h5>
    major: Failure in tools library
    minor: error in function
  #001: C:\autotest\hdf5112-StdRelease-dist-10vs16\build\hdfsrc\tools\src\h5repack\h5repack_copy.c line 1110 in do_copy_objects(): can't allocate space for hyperslab
    major: Failure in tools library
    minor: error in function

Seeing as setenv H5TOOLS_BUFSIZE 536870912 would set the buffer size to 512 TB rather than 512 MB, the allocation error is to be expected.

[byrnHDF]

Yes, the environment variable, H5TOOLS_BUFSIZE, should be set as the number of megabytes (var * 1024 * 1024) but the tools variable, H5TOOLS_BUFSIZE, is the number of bytes.

[byrnHDF]

Addressed this in #4602 with a Note about the ENV variable, H5TOOLS_BUFSIZE, and an an example.