colord icon indicating copy to clipboard operation
colord copied to clipboard

An Exception Test

Open fahaihi opened this issue 10 months ago • 1 comments

Dear CoLoRd developer. We used CoLoRd for FastQ Long Reads' no-reference compression experiment. In the dataset ERR11011595(https://www.ebi.ac.uk/ena/browser/view/ERR11011595) , run the following command: /bin/time -v -p colord compress-ont -q org -p ratio -t 16 ERR11011595.fastq ERR11011595.colord . We measured memory and time using the /bin/time -v -p instruction, and the result was a compression time of up to 45.521 hours, while the dataset size was only 4.411 GB, which is not consistent with our understanding of CoLoRd's superior compression performance. Do you know what the problem is...? TKU!

fahaihi avatar Aug 31 '23 02:08 fahaihi

Hello! Thank you very much for reporting this. There were indeed performance-related issue in the code. It was caused because of some characteristics of this particular dataset. I think I mostly fixed it with 13e8e94ab86924a49c0d49feaa0cd49af41f1e39 There is a new release published . On our server in the default mode with 16 threads colors compressed this dataset in ~5 min 30 sec. I think there may still be some room for improvement in the case of this dataset, but for now, we have this :)

Thank you again for reporting this. Let us know how it works on your end now, and let me know if I may close this issue (or close it yourself).

Best Marek

marekkokot avatar Sep 11 '23 13:09 marekkokot