Stuck on Alignment using --gpu

[JustinBellavance]

Hello,

I'm encountering a problem using gem-mapper on --gpu mode. The code runs fine until it gets to [PE::Mapping Sequences], where it gets stuck with no further output. It is not getting killed or sending an error message. I'm using a gpu-index. Interestingly, the header of the .sam gets outputted, but not the alignment itself.

The same command works well when --gpu is not included, using a cpu index.

Here is the full command:

gem-mapper -I Homo_sapiens.GRCh38.gem -z -1 ${sample}.trim.pair1.fastq.gz -z -2 ${sample}.trim.pair2.fastq.gz --gpu -t 16

I'm using an HPC environment, where the code is being run with either an NVidia Tesla V100-SXM2-16GB or NVidia A100 (40 GB memory). For the former, we're getting this warning: WARNING PERF. SLOWDOWNS (Mem Recommended: 21065 MBytes - Mem Available: 15059 MBytes).

Using htop, it seems that memory and cpu were being utilized. Using nvidia-smi, I saw that the GPU-utilization was close to 100%, but the power usage was very low compared to other programs (~70 W / 300 W).

Would really appreciate any help, thanks!

[smarco]

Hello,

I'm sorry to hear that you encountered this problem. The GEM3 GPU support was developed by @achacond and AFAIK he continued its development beyond the GPU version that can be found in this repo. I believe he has a more updated (and efficient) version under his repo achacond-gem3-mapper. So, I would first try his version and see if the problem disappears. If the problem persists, the best I can do is to recommend using the regular CPU version, which I implemented and, thus, I know better.

Also, perhaps, @achacond can comment on this issue if he has time.

Hope this helps.

[Luobiny]

I ran into the same issue with --gpu enabled on a machine with GPUs. It just hangs there without generating any output nor updating log messages.