samtools
bcftools concat retry mechanisms?
For the bcftools concat function, could we add some retry mechanisms when directly accessing s3 files? I am using bcftools concat to directly concatenate a large number of vcf files. Due to the huge volume, if there are network fluctuations during the concatenation process, the connection is prone to aborting halfway, resulting in a "software caused connection abort" error. This makes it difficult to successfully complete the concatenation. Would it be possible to add some retry mechanisms, such as retrying three times, which could significantly improve this situation of concatenation failures?
Currently, I am unable to reproduce this issue, presumably because the network fluctuations have subsided. Unfortunately, I didn't save the log. I recall that before the "software caused connection abort" error, there was an "[E::hts_hopen]" error
I can imagine this, probably best done at htslib level and maybe controlled by an environment variable.
Just to link it to a related issue in htslib: https://github.com/samtools/htslib/issues/1424
I reproduce it, the log is [E::hts_open_format] Failed to open file "s3://xxxxx" : Software caused connection abort