pigz
pigz copied to clipboard
madler
MPI support
Hi, I'm a complete noob in MPI parallel programming and I can't stop the impression that this environment lacks software for daily, CPU-intensive tasks, such as file compression. If and when I get up to a point when I'm comfortable with MPI libraries, I'd gladly dive into implementing this feature. However, I was curious whether the approach of spreading the compression tasks over multiple machines in LAN has been tried in the past, and maybe it's been already proven to be pointless? Any thoughts would be much appreciated.
Based on my experience with a competing project (create_compressed_fs from cloop-utils) it can be quite beneficial but you need to make sure that the pipeline is constantly filled, the latencies are reduced as far as possible (TCP_NODELAY, TCP_QUICKACK) and the network performs very well in general.
I understand why high throughput would be beneficial, but low latency? Hm, not sure why would that help, as long as CPUs are the actual bottleneck of the whole process. Indeed each thread should have its own task buffer to stay busy around the clock.
MPI parallel compression might not be needed. When using zlib-ng as zlib library, compression speed is 4 times faster: https://github.com/zlib-ng/zlib-ng/issues/908
