hdf5-blosc icon indicating copy to clipboard operation
hdf5-blosc copied to clipboard

Published 20 hours ago • verified publisher icon Blosc

h5repack with blosc multicore compression

Open Castellor opened this issue 5 years ago • 2 comments

I'm using hdf5-1.10.5 on windows with installed hdf5plugins-1.10.5 and tried to set the env BLOSC_NTHREADS = 10 but only one core is used.

I want to compress images with CHUNK=1x140x160.

What else has to be done to get multicore support?

Castellor avatar May 03 '19 18:05 Castellor

With just 22400 items, your chunk seems too small for blosc to split it in different blocks (the only way to leverage multithreading). Try with something that is at least 512 KB or more (up to your L3 cache size, typically 4 MB or 8 MB).

FrancescAlted avatar May 04 '19 08:05 FrancescAlted

ok thanks, but maybe I have a more fundamental problem. The download from https://www.hdfgroup.org/downloads/hdf5/ with the plugins reads "hdf5plugin-1.10.5_decode-win10_64-vs15", so I have to build my own with encoding enabled, right? so i have to build (Windows 10):

  • "hdf5-blosc" and "c-blosc" or "c-blosc2"

Castellor avatar May 07 '19 08:05 Castellor