bitshuffle icon indicating copy to clipboard operation
bitshuffle copied to clipboard

Lock free decompression

Open jonwright opened this issue 3 years ago • 4 comments

As mentioned in issue #138, you can get a speed up for certain datasets on large CPU's by using a static openmp schedule. In this pull request, all threads read all offsets to find the compressed data and output pointers are computed.

Depending on whether load balancing is needed by other people, it might be better to use schedule(runtime) instead of schedule(static).

jonwright avatar Feb 06 '23 15:02 jonwright

Just coming back to this after the person who maintained this package for the last few years has moved on to other things. This looks very cool. Could you update this branch with everything that's been added in the interim? That should trigger the continuous integration.

kiyo-masui avatar Sep 24 '24 14:09 kiyo-masui

So far I only clicked "update branch" on github - presumably it will need some more work before it will be ready.

jonwright avatar Sep 24 '24 15:09 jonwright

Tests fail with a "bus error 10" which sounds like some sort of issue addressing memory.

kiyo-masui avatar Sep 24 '24 15:09 kiyo-masui

Sorry for that - the init was missed for non-openmp builds - it should be ready for you to look at now

jonwright avatar Sep 25 '24 19:09 jonwright