What_the_Phage icon indicating copy to clipboard operation
What_the_Phage copied to clipboard

deepvirfinder uses more cores unexpectly

Open Microbion opened this issue 1 year ago • 3 comments

--max_cores parameter can't limit deepvirfinder, because the theano backend to keras automatically uses all available cores for the internal matrix operations. as mentioned here https://github.com/jessieren/DeepVirFinder/issues/4 I wonder other tools have the similar issue, such as a numpy issue mentioned here https://stackoverflow.com/questions/30791550/limit-number-of-threads-in-numpy. Is it necessary and possible to add some environment variables for avoid using all cores?

Microbion avatar Jul 21 '23 00:07 Microbion

I think you have to ask the phage prediction tool developer directly as I cannot adjust the code of these tools

Are these commands you would use in a bash terminal before executing e.g. deepvirfinder? export MKL_NUM_THREADS=1 If yes I think its possible then

mult1fractal avatar Jul 25 '23 07:07 mult1fractal

I modified line 10 in workflows/process/deepvirfinder/deepvirfinder.wf, by OMP_NUM_THREADS=${task.cpus} dvf.py -c ${task.cpus} -i ${fasta} -o ${name}, and it worked. Consider of some tools' can't limit resource by itself or can but not correctly (not just deepvirfinder, I meeted similar troubles in other tools), I think it's necessary to limit resource manurally to improve robustness of workflow.

Microbion avatar Jul 25 '23 13:07 Microbion

nice! by chance do you know which tools have this issue?

I would update the code accordingly and create a new release candidate

mult1fractal avatar Jul 26 '23 13:07 mult1fractal