zstd icon indicating copy to clipboard operation
zstd copied to clipboard

seekable_format/examples/parallel_compression.c is not parallel

Open EbiArnie opened this issue 1 year ago • 7 comments

Describe the bug The parallel compression example does not seem to be working in parallel. It runs on a single thread only, regardless of how many threads are requested.

To Reproduce

  1. Build the example parallel_compression
  2. Run: ./parallel_compression data.txt $((2*1024*1024)) 4 data.txt can be any sufficiently large file so that compression takes a handful of seconds. In my case, this is ASCII data, 2.8 GB.
  3. While the compression is running, look at e.g. top
  4. Just a single thread is used. This is usually slower than the single threaded seekable_compression,

Expected behavior Parallel execution should use multiple cores and be faster than single threaded. The full zstd executable works as expected; given the -T option it is faster than without and saturates all cores.

Also, I'd love for --seekable to become a standard option for zstd, but that should probably be a separate issue.

Desktop:

  • OS: Linux
  • Version Ubuntu 22.04
  • Compiler: gcc 11.4
  • Flags: -O3
  • System: Quad-core Intel Core i7-1165G7
  • Build system Makefile
  • Built from git version: commit 559762da12f54712d44f619098aa4a7e7bc5727b, Mar 14 2024

EbiArnie avatar Mar 19 '24 10:03 EbiArnie