inikep
multi-threading option
We may add a support for multi-threading to lzbench in the future but it will be a new feature. Looking for volunteers :)
Had a look how is it organised and I thought - Is there a way to simply pass an argument to the de/compressor? Maybe, instead of multiplying arguments of function, like param1, param2... paramN
inline int64_t lzbench_compress(lzbench_params_t *params, std::vector<size_t>& chunk_sizes, compress_func compress, std::vector<size_t> &compr_sizes, uint8_t *inbuf, uint8_t *outbuf, size_t outsize, size_t param1, size_t param2, char* workmem);
aggregate all those additional, optional parameters in a struct and pass a struct to the function.
inline int64_t lzbench_compress(lzbench_params_t *params, std::vector<size_t>& chunk_sizes, compress_func compress, std::vector<size_t> &compr_sizes, uint8_t *inbuf, uint8_t *outbuf, size_t outsize, struct options *codec_params);
struct codec_options { param1; param2; (...) paramN; };
It would allow to seamless extension in the future as only thing passed would be a pointer (to struct).
Also that could require to make new column called 'threads' that would be turn on in case of threads option to the program. Just to give justice to that codec and the others that don't necessarily use threads. Well, at least to show it after header.
it's much easier to reach memory or storage limits
It the user's choice.
what skews results
Like how?
only modern compressors support multithreading
And modern hardware, which is likely to ba able to handle load and requirements of the task. Again - user's choice.
should we compare e.g. a single threaded lzf with multithreaded lz4?
If it's clearly stated it multi-threaded result it's user's choice.
even if a compressor support multithreading we will compare not just compressors but also how effective is its multi-threading implementation vs other implementations
It would only use 'internal', 'codec's implementation'. Again - user's choice.
Most, if not all, multi-threaded compressors I can think of operate on client not codec level. And I think it's a good thing. I wouldn't worry about that here. If codec offers multi-threading, it's their job to do it efficiently.
to make testing fair we can implement multi-threading in lzbench for all compressors and disable internal multi-threading implementation (but this is a new feature)
Again, if it was clearly marked as X-threaded, then it's user's decision to make conclusions.
All in all, allowing 'codec's internal' threading is ok as long as it's clearly stated it's multi-threaded measure.
aggregate all those additional, optional parameters in a struct and pass a struct to the function
We can do it
Like how?
Normally our results are CPU-bound. With multi-threading the results may be I/O-bound so on the same machine you will get different results depending on storage you are using.
It would only use 'internal', 'codec's implementation'.
Some MT implementation are done outside the compression library and it will be hard to use them.
For example it seems that MT for LZ4 is implemented in programs:
https://github.com/inikep/lzbench/blob/master/lz4/programs/lz4io.c
https://github.com/inikep/lzbench/blob/master/lz4/programs/threadpool.c
Moreover we will come back to situation of kanzi MT vs lzf single-threaded. I know, it will be described, but still not fair IMHO :)
With multi-threading the results may be I/O-bound
I thought the whole corpus is read into memory. There is no 'I/O' in memory, is it?
Some MT implementation are done outside the compression library
Only pbzip2, lbzip2, pigz, plzip, bsc(?), bzip3, zstd-mt, rar (I guess) that's all I can think of, but for sure, I miss some more.
Those 'internal' I know of libbsc(?), libsais, fast-lzma2, and now kanzi.
I can see a pattern here. If you wold implement MT then it would be 'external', 'client-thing', and the bench would test implementation, rather than codec. I wouldn't go this way.
Moreover we will come back to situation of kanzi MT vs lzf single-threaded. I know, it will be described, but still not fair IMHO
If it was clearly stated that it's multi-threaded, and how multi-threaded, then it's not a problem.
Could look like this: regular result:
$ lzbench -ekazni ./lzbench
lzbench 2.0.1 (64-bit Linux)
Compressor name Compress. Decompress. Compr. size Ratio Filename
memcpy 11612 MB/s 11732 MB/s 7615680 100.00 ./lzbench
kanzi 2.3 -2 19.7 MB/s 80.6 MB/s 3272575 42.97 ./lzbench
kanzi 2.3 -3 12.9 MB/s 83.1 MB/s 3262039 42.83 ./lzbench
kanzi 2.3 -4 5.28 MB/s 20.3 MB/s 3104334 40.76 ./lzbench
Then user specifies threaded option: 1 thread:
$ lzbench -T1 -ekazni ./lzbench
lzbench 2.0.1 (64-bit Linux)
Using threads: 1, block size: x
Compressor name Th Compress. Decompress. Compr. size Ratio Filename
memcpy 1 11612 MB/s 11732 MB/s 7615680 100.00 ./lzbench
kanzi 2.3 -2 1 19.7 MB/s 80.6 MB/s 3272575 42.97 ./lzbench
kanzi 2.3 -3 1 12.9 MB/s 83.1 MB/s 3262039 42.83 ./lzbench
kanzi 2.3 -4 1 5.28 MB/s 20.3 MB/s 3104334 40.76 ./lzbench
2 threads:
$ lzbench -T2 -ekazni ./lzbench
lzbench 2.0.1 (64-bit Linux)
Using threads: 2, block size: x
Compressor name Th Compress. Decompress. Compr. size Ratio Filename
memcpy 1 11612 MB/s 11732 MB/s 7615680 100.00 ./lzbench
kanzi 2.3 -2 2 30.6 MB/s 108 MB/s 3264768 42.87 ./lzbench
kanzi 2.3 -3 2 16.6 MB/s 84.1 MB/s 3256681 42.77 ./lzbench
kanzi 2.3 -4 2 6.19 MB/s 25.6 MB/s 3099558 40.70 ./lzbench
'block size' is optional parameter to threads. Obviously it's not provided in the example, nor used by kanzi. Just thought it could be one of parameters to threads. Doesn't have to be.
Could also be limited to one codec only, when using threads. That would solve the problem of 'fairness'.
I thought the whole corpus is read into memory. There is no 'I/O' in memory, is it?
Agree. I got confused with database benchmarks :)
Those 'internal' I know of libbsc(?), libsais, fast-lzma2, and now kanzi.
Also zstd but not so many of them.
Those 'internal' I know of libbsc(?), libsais, fast-lzma2, and now kanzi.
Also zstd but not so many of them.
Zstd library uses threads internally? I thought it was implemented in client.
FYI, there is another one that uses threads.
It's in the lib:
https://github.com/inikep/lzbench/blob/master/zstd/lib/compress/zstd_compress.c#L6354-L6406
but you have to turn it on with -DZSTD_MULTITHREAD.
Unreleased yet lzbench 2.1 (please help with testing "master" branch at https://github.com/inikep/lzbench) introduces parallel compression: https://encode.su/threads/2579-lzbench-in-memory-benchmark-of-open-source-compressors?p=85594&viewfull=1#post85594
I have tried to separate these features so compression, decompression and codec threading are separete. First attempt tried to parse input and get them separately. By the way I thought up way to all three options under one command line switch. Then, second take tried to make it actually work. Well, there is still a problem with a codec threads. Maybe it should land in a new feature branch.
Lzbench creates a thread pool with T threads at the begining and reuses it for compression and decompression with all selected codecs. To use different number of threads for compression and decompression will require to use 2 thread pools or rewrite current thread pool (harder to do).
https://github.com/inikep/lzbench/commits/decomp_threads/ implements support for -Tc,d where c = compression threads, d = decompression threads
I do not insist on separate compression and decompression thread pools. I only observed this original behaviour where it uses different compression and decompression threads on its own, automatically.
$ ./lzbench-T -T4 -ezlib,1/bzip2,9 dickens
lzbench 2.1 | GCC 6.5.0 | 32-bit Linux
Compressor name C,D Threads Compress. Decompress. Compr. size Ratio Filename
zlib 1.3.1 -1 3, 4 23.4 MB/s 103 MB/s 4585618 44.99 dickens
bzip2 1.0.8 -9 1, 2 4.63 MB/s 8.71 MB/s 2799520 27.47 dickens
[Summary] Files=1 Chunks=1 Threads=4
[Params] cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB
Also I cared more about giving chance to use internal codec threads - therefore optional third argument to `-T' indicating codec threads.
It doesn't brace to be third argument and C and D don't have to ne separate, but it could be as last argument to `T'.
Either -TC,K or -TC,D,K, where:
C - compression threads, D - decompression threads, K - internal codec threads.
C,D Threads shows how many threads were used by threadpool. But it shows the total number for all iterations:
$ ./lzbench -T4 -ezlib,1/bzip2,9 ../silesia/dickens
lzbench 2.1.1 | GCC 14.2.0 | 64-bit Linux | AMD EPYC 9554 64-Core Processor
Compressor name C,D Threads Compress. Decompress. Compr. size Ratio Filename
zlib 1.3.1 -1 4, 4 94.1 MB/s 203 MB/s 4585618 44.99 ../silesia/dickens
bzip2 1.0.8 -9 2, 4 16.6 MB/s 36.8 MB/s 2799520 27.47 ../silesia/dickens
[Summary] Files=1 Chunks=1 cThreads=4 dThreads=4
[Params] cIters=1 dIters=1 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB
If you want to see numbers for a single iteration you have to use -t0,0:
$ ./lzbench -T4 -t0,0 -ezlib,1/bzip2,9 ../silesia/dickens
lzbench 2.1.1 | GCC 14.2.0 | 64-bit Linux | AMD EPYC 9554 64-Core Processor
Compressor name C,D Threads Compress. Decompress. Compr. size Ratio Filename
zlib 1.3.1 -1 1, 1 88.5 MB/s 202 MB/s 4585618 44.99 ../silesia/dickens
bzip2 1.0.8 -9 1, 1 16.2 MB/s 33.4 MB/s 2799520 27.47 ../silesia/dickens
[Summary] Files=1 Chunks=1 cThreads=4 dThreads=4
[Params] cIters=1 dIters=1 cTime=0.0 dTime=0.0 chunkSize=1706MB cSpeed=0MB
Results with 8 internal threads:
$ ./lzbench -t0,0 -eint_mt -I8 ../silesia.tar
lzbench 2.1 | GCC 14.2.0 | 64-bit Linux | AMD EPYC 9554 64-Core Processor
Compressor name I_Threads Compress. Decompress. Compr. size Ratio Filename
memcpy 8 15883 MB/s 16230 MB/s 211947520 100.00 ../silesia.tar
fastlzma2 1.0.1 -1 8 35.4 MB/s 403 MB/s 59573883 28.11 ../silesia.tar
fastlzma2 1.0.1 -5 8 36.8 MB/s 569 MB/s 51327104 24.22 ../silesia.tar
fastlzma2 1.0.1 -10 8 23.9 MB/s 107 MB/s 48668818 22.96 ../silesia.tar
kanzi 2.3 -1 8 646 MB/s 1666 MB/s 80280672 37.88 ../silesia.tar
kanzi 2.3 -2 8 570 MB/s 1384 MB/s 68264304 32.21 ../silesia.tar
kanzi 2.3 -3 8 441 MB/s 1030 MB/s 64963864 30.65 ../silesia.tar
kanzi 2.3 -4 8 281 MB/s 615 MB/s 60767201 28.67 ../silesia.tar
kanzi 2.3 -5 8 125 MB/s 258 MB/s 54050463 25.50 ../silesia.tar
kanzi 2.3 -6 8 81.2 MB/s 147 MB/s 49517568 23.36 ../silesia.tar
kanzi 2.3 -7 8 53.7 MB/s 77.8 MB/s 47308205 22.32 ../silesia.tar
lzham 1.0 -d26 -0 8 6.77 MB/s 284 MB/s 64255692 30.32 ../silesia.tar
lzham 1.0 -d26 -4 8 2.49 MB/s 390 MB/s 51150615 24.13 ../silesia.tar
lzma 24.09 -0 8 282 MB/s 471 MB/s 60840494 28.71 ../silesia.tar
lzma 24.09 -4 8 37.6 MB/s 259 MB/s 55968602 26.41 ../silesia.tar
lzma 24.09 -9 8 4.46 MB/s 102 MB/s 48683275 22.97 ../silesia.tar
xz 5.8.1 -0 8 218 MB/s 114 MB/s 63042292 29.74 ../silesia.tar
xz 5.8.1 -4 8 33.2 MB/s 142 MB/s 52458204 24.75 ../silesia.tar
xz 5.8.1 -9 8 3.07 MB/s 145 MB/s 48766532 23.01 ../silesia.tar
zstd 1.5.7 -1 8 1502 MB/s 1585 MB/s 73292642 34.58 ../silesia.tar
zstd 1.5.7 -5 8 802 MB/s 1104 MB/s 62847016 29.65 ../silesia.tar
zstd 1.5.7 -9 8 375 MB/s 1136 MB/s 59179159 27.92 ../silesia.tar
zstd 1.5.7 -14 8 56.0 MB/s 1172 MB/s 57396270 27.08 ../silesia.tar
zstd 1.5.7 -18 8 18.9 MB/s 1072 MB/s 53313874 25.15 ../silesia.tar
zstd 1.5.7 -22 8 2.27 MB/s 989 MB/s 52288411 24.67 ../silesia.tar
[Params] cIters=1 dIters=1 cTime=0.0 dTime=0.0 chunkSize=0KB cSpeed=0MB
Results with 1 thread:
$ ./lzbench -t0,0 -eint_mt ../silesia.tar
lzbench 2.1 | GCC 14.2.0 | 64-bit Linux | AMD EPYC 9554 64-Core Processor
Compressor name Compress. Decompress. Compr. size Ratio Filename
memcpy 15961 MB/s 16082 MB/s 211947520 100.00 ../silesia.tar
fastlzma2 1.0.1 -1 51.0 MB/s 1351 MB/s 59590594 28.12 ../silesia.tar
fastlzma2 1.0.1 -5 56.0 MB/s 531 MB/s 51890964 24.48 ../silesia.tar
fastlzma2 1.0.1 -10 49.3 MB/s 108 MB/s 48757107 23.00 ../silesia.tar
kanzi 2.3 -1 214 MB/s 1387 MB/s 80280672 37.88 ../silesia.tar
kanzi 2.3 -2 187 MB/s 521 MB/s 68264304 32.21 ../silesia.tar
kanzi 2.3 -3 128 MB/s 387 MB/s 64963864 30.65 ../silesia.tar
kanzi 2.3 -4 65.0 MB/s 186 MB/s 60767201 28.67 ../silesia.tar
kanzi 2.3 -5 25.4 MB/s 60.5 MB/s 54050463 25.50 ../silesia.tar
kanzi 2.3 -6 18.9 MB/s 35.2 MB/s 49517568 23.36 ../silesia.tar
kanzi 2.3 -7 12.2 MB/s 18.0 MB/s 47308205 22.32 ../silesia.tar
lzham 1.0 -d26 -0 13.9 MB/s 284 MB/s 64089870 30.24 ../silesia.tar
lzham 1.0 -d26 -4 1.78 MB/s 409 MB/s 51012554 24.07 ../silesia.tar
lzma 24.09 -0 37.2 MB/s 88.4 MB/s 60520844 28.55 ../silesia.tar
lzma 24.09 -4 14.1 MB/s 101 MB/s 55936064 26.39 ../silesia.tar
lzma 24.09 -9 2.83 MB/s 103 MB/s 48682482 22.97 ../silesia.tar
xz 5.8.1 -0 28.2 MB/s 114 MB/s 63042292 29.74 ../silesia.tar
xz 5.8.1 -4 5.83 MB/s 141 MB/s 52458204 24.75 ../silesia.tar
xz 5.8.1 -9 2.97 MB/s 144 MB/s 48766532 23.01 ../silesia.tar
zstd 1.5.7 -1 493 MB/s 1590 MB/s 73193704 34.53 ../silesia.tar
zstd 1.5.7 -5 147 MB/s 1391 MB/s 62740852 29.60 ../silesia.tar
zstd 1.5.7 -9 72.4 MB/s 1494 MB/s 59071700 27.87 ../silesia.tar
zstd 1.5.7 -14 13.2 MB/s 1576 MB/s 57274741 27.02 ../silesia.tar
zstd 1.5.7 -18 4.45 MB/s 1394 MB/s 53288948 25.14 ../silesia.tar
zstd 1.5.7 -22 2.21 MB/s 1281 MB/s 52284290 24.67 ../silesia.tar
[Params] cIters=1 dIters=1 cTime=0.0 dTime=0.0 chunkSize=0KB cSpeed=0MB
@tansy At https://github.com/inikep/lzbench/tree/internal-codec-threads I added all codecs that support internal multi-threading.
fastlzma2 and lzham doesn't work well with MT on my machine but the rest look OK.
Results with 1 thread:
Fastlzma2, lzham decompression results are suspiciously high, and higher with lower compression levels, which is illogical. Compare it to xz, or lzlib and you will see what happens to decompression speed with compression.
CI test before threading introduction looks like this:
lzma 24.09 -0 18.7 MB/s 39.8 MB/s 2187301 38.13 ./lzbench
lzma 24.09 -2 13.2 MB/s 42.5 MB/s 2136695 37.25 ./lzbench
lzma 24.09 -4 9.51 MB/s 43.1 MB/s 2131399 37.15 ./lzbench
lzma 24.09 -6 3.12 MB/s 44.6 MB/s 1955231 34.08 ./lzbench
lzma 24.09 -9 2.87 MB/s 44.7 MB/s 1950906 34.01 ./lzbench
fastlzma2 1.0.1 -1 12.7 MB/s 40.2 MB/s 2145730 37.40 ./lzbench
fastlzma2 1.0.1 -3 6.53 MB/s 42.3 MB/s 2042504 35.60 ./lzbench
fastlzma2 1.0.1 -5 5.22 MB/s 43.4 MB/s 2017011 35.16 ./lzbench
fastlzma2 1.0.1 -8 4.35 MB/s 43.4 MB/s 1965904 34.27 ./lzbench
fastlzma2 1.0.1 -10 3.91 MB/s 43.4 MB/s 1968089 34.31 ./lzbench
lzham 1.0 -d26 -0 6.13 MB/s 108 MB/s 2260437 39.40 ./lzbench
lzham 1.0 -d26 -1 2.48 MB/s 113 MB/s 2119862 36.95 ./lzbench
Decompression speed increases slightly, and more/less proportionally, with compression, and is not 10+ times for lower compression levels.
They must 'secretly' use multiple threads under the hood. I don't see any other explanation.
PS. last CI test is analogical. It stems from the fact that CI tests are done in virtual machine that is given one processor. So they can be treated as truly single-threaded. There is a catch though - in case of multithreded test it will nor be any faster and will nor indicate threads work this way.
The results from master branch show that lzham is OK but fastlzma2 uses threads when it shouldn't:
lzbench 2.1 | GCC 14.2.0 | 64-bit Linux | AMD EPYC 9554 64-Core Processor
Compressor name Compress. Decompress. Compr. size Ratio Filename
memcpy 16023 MB/s 16354 MB/s 211947520 100.00 ../silesia.tar
fastlzma2 1.0.1 -1 26.5 MB/s 91.7 MB/s 59030950 27.85 ../silesia.tar
fastlzma2 1.0.1 -5 8.70 MB/s 105 MB/s 51209567 24.16 ../silesia.tar
fastlzma2 1.0.1 -10 3.73 MB/s 108 MB/s 48666061 22.96 ../silesia.tar
kanzi 2.3 -1 214 MB/s 1389 MB/s 80280672 37.88 ../silesia.tar
kanzi 2.3 -2 189 MB/s 522 MB/s 68264304 32.21 ../silesia.tar
kanzi 2.3 -3 128 MB/s 388 MB/s 64963864 30.65 ../silesia.tar
kanzi 2.3 -4 64.7 MB/s 185 MB/s 60767201 28.67 ../silesia.tar
kanzi 2.3 -5 25.3 MB/s 61.0 MB/s 54050463 25.50 ../silesia.tar
kanzi 2.3 -6 19.0 MB/s 37.2 MB/s 49517568 23.36 ../silesia.tar
kanzi 2.3 -7 12.2 MB/s 18.0 MB/s 47308205 22.32 ../silesia.tar
lzham 1.0 -d26 -0 14.1 MB/s 283 MB/s 64089870 30.24 ../silesia.tar
lzham 1.0 -d26 -4 1.82 MB/s 411 MB/s 51012554 24.07 ../silesia.tar
lzma 24.09 -0 37.2 MB/s 89.4 MB/s 60509826 28.55 ../silesia.tar
lzma 24.09 -4 14.2 MB/s 102 MB/s 55926363 26.39 ../silesia.tar
lzma 24.09 -9 2.83 MB/s 111 MB/s 48682007 22.97 ../silesia.tar
xz 5.8.1 -0 28.4 MB/s 118 MB/s 62579435 29.53 ../silesia.tar
xz 5.8.1 -4 5.53 MB/s 146 MB/s 52106396 24.58 ../silesia.tar
xz 5.8.1 -9 2.96 MB/s 147 MB/s 48745306 23.00 ../silesia.tar
zstd 1.5.7 -1 499 MB/s 1593 MB/s 73193704 34.53 ../silesia.tar
zstd 1.5.7 -5 148 MB/s 1386 MB/s 62740852 29.60 ../silesia.tar
zstd 1.5.7 -9 72.3 MB/s 1508 MB/s 59071700 27.87 ../silesia.tar
zstd 1.5.7 -14 13.3 MB/s 1577 MB/s 57274741 27.02 ../silesia.tar
zstd 1.5.7 -18 4.42 MB/s 1392 MB/s 53288948 25.14 ../silesia.tar
zstd 1.5.7 -22 2.20 MB/s 1267 MB/s 52284290 24.67 ../silesia.tar
[Params] cIters=1 dIters=1 cTime=0.0 dTime=0.0 chunkSize=0KB cSpeed=0MB
Well, it looks like fastlzma2 in your last test is ok - around 100 MB/s decompression, on par with lzma, and increases with compression, which is normal too.
I fixed fastlzma2 and updated https://github.com/inikep/lzbench/commits/internal-codec-threads/
@tansy I just merged to master:
- improvement: Added the -I# option, which enables compression using # internal (built-in) threads
when supported. Currently available for the following compressors: fast-lzma2,
kanzi, lzham, lzma, xz, and zstd.
Please check if everything works fine for you.
How about adding threading support to codec list? I tried this thing and it's somewhat useful if you want to find out which codec supports threads. Not sure if it should look this way, but that's visual detail. There can be more details if necessary.
$ ./lzbench -vl
Available compressors for -e option:
memcpy = memcpy
brieflz = brieflz 1.3.0 [1-9]
...
fastlz = fastlz 0.5.0 [1-2]
fastlzma2 = fastlzma2 1.0.1 [1-10] th
gipfeli = gipfeli 2016-07-13
...
glza = glza 0.11.8
kanzi = kanzi 2.4.0 [1-9] th
libdeflate = libdeflate 1.24 [1-12]
...
lzg = lzg 1.0.10 [1-9]
lzham = lzham 1.0 -d26 [0-4] th
lzham22 = lzham 1.0 -d22 [0-4] th
lzham24 = lzham 1.0 -d24 [0-4] th
lzjb = lzjb 2010
lzlib = lzlib 1.15 [0-9]
lzma = lzma 25.01 [0-9] th
lzmat = lzmat 1.01
I implemented this but it looks overwhelming:
memcpy = memcpy, threading=external_pool
brieflz = brieflz 1.3.0; supported levels=[1-9], threading=external_pool
brotli = brotli 1.1.0; supported levels=[0-11], threading=external_pool
brotli22 = brotli 1.1.0 -d22; supported levels=[0-11], threading=external_pool
brotli24 = brotli 1.1.0 -d24; supported levels=[0-11], threading=external_pool
bsc0 = bsc 3.3.11 -m0 -e2, threading=external_pool
bsc1 = bsc 3.3.11 -m0 -e1, threading=external_pool
bsc2 = bsc 3.3.11 -m0 -e0, threading=external_pool
bsc3 = bsc 3.3.11 -m3 -e1, threading=external_pool
bsc4 = bsc 3.3.11 -m4 -e1, threading=external_pool
bsc5 = bsc 3.3.11 -m5 -e1, threading=external_pool
bsc6 = bsc 3.3.11 -m6 -e1, threading=external_pool
bzip2 = bzip2 1.0.8; supported levels=[1-9], threading=external_pool
bzip3 = bzip3 1.5.2; supported levels=[1-10], threading=external_pool
crush = crush 1.0; supported levels=[0-2], threading=none
csc = csc 2016-10-13; supported levels=[1-5], threading=external_pool
density = density 0.16.6; supported levels=[1-3], threading=external_pool
fastlz = fastlz 0.5.0; supported levels=[1-2], threading=external_pool
fastlzma2 = fastlzma2 1.0.1; supported levels=[1-10], threading=internal,external_pool
gipfeli = gipfeli 2016-07-13, threading=external_pool
glza = glza 0.11.8, threading=none
This looks better:
memcpy = memcpy; threading=-T
brieflz = brieflz 1.3.0; levels=[1-9]; threading=-T
brotli = brotli 1.1.0; levels=[0-11]; threading=-T
brotli22 = brotli 1.1.0 -d22; levels=[0-11]; threading=-T
brotli24 = brotli 1.1.0 -d24; levels=[0-11]; threading=-T
bsc0 = bsc 3.3.11 -m0 -e2; threading=-T
bsc1 = bsc 3.3.11 -m0 -e1; threading=-T
bsc2 = bsc 3.3.11 -m0 -e0; threading=-T
bsc3 = bsc 3.3.11 -m3 -e1; threading=-T
bsc4 = bsc 3.3.11 -m4 -e1; threading=-T
bsc5 = bsc 3.3.11 -m5 -e1; threading=-T
bsc6 = bsc 3.3.11 -m6 -e1; threading=-T
bzip2 = bzip2 1.0.8; levels=[1-9]; threading=-T
bzip3 = bzip3 1.5.2; levels=[1-10]; threading=-T
crush = crush 1.0; levels=[0-2]; threading=none
csc = csc 2016-10-13; levels=[1-5]; threading=-T
density = density 0.16.6; levels=[1-3]; threading=-T
fastlz = fastlz 0.5.0; levels=[1-2]; threading=-T
fastlzma2 = fastlzma2 1.0.1; levels=[1-10]; threading=-I,-T
gipfeli = gipfeli 2016-07-13; threading=-T
glza = glza 0.11.8; threading=none
`threading=-T' is redundant. They all support it and none of them does, in the same time. It's like multithreading in lbzip2/plzip - it's done by client not by compressor library itself, so only internal threading is relevant for codec.
The list shows what options are supported by given compressors in lzbench. For different reasons -T is disabled for crush, glza, tornado, yappy.
The list shows what options are supported by given compressors in lzbench.
Exactly. `-T' is provided by the bench not by the codec, so it has nothing to do with codec. If these are details pertaining codec then out had nothing to do with the codec, therefore no need to display it in codec's properties. Plus, If you display it for every codec, no wonder it 'looks overwhelming'.
For different reasons -T is disabled for crush, glza, tornado, yappy.
And that is relevant information, relevant to the codec itself, not to the application as a.whole.
It doesn't matter if an option is provided by bench or codec itself. I wan't to know what options are supported for each codec.
But OK, let's start without threading=-T: https://github.com/inikep/lzbench/commit/5132e6ddc3
Pool threading is supported for every codec, so it's not any specific to this or that codec. I'll think of it yet.