Merge speed has not been increased by '--threads' option ?

[Jerry-Wang-Dog]

bcftools merge ---file-list list.txt -O z -o output.vcf --threads 16

Why the CPU% is only about 130%, but Virtual Memory has increased to 60G？ What‘s important is that the MERGE speed has not been increased！！！

（My file-list contains 200 samples with vcf.gz format. There are infomation of each site in those VCF files, not only variants site）

[davmlaw]

Hi, it might help to run some benchmarks with and without threads recording run time, CPU, memory and I/O. This provides context to "increased to 60G" (ie how much ram it used before)

/usr/bin/time --verbose bcftools merge ---file-list  list.txt -O z -o output.vcf  2> benchmark_no_threads.txt
/usr/bin/time --verbose bcftools merge ---file-list  list.txt -O z -o output.vcf --threads 16 2> benchmark_threads.txt

You could also try 2/4/8 threads and see what happens there

[jcm6t]

Per the bcftools manual, Common Options

--threads INT
Use multithreading with INT worker threads. The option is currently used only for the compression of the output stream, only when --output-type is b or z. Default: 0.