distiller-nf icon indicating copy to clipboard operation
distiller-nf copied to clipboard

how memory limit gets specified in cluster using

Open ryao-mdanderson opened this issue 2 years ago • 0 comments

Dear distiller-nf support,

This is a beginner question. I am using the test example on an HPC cluster RHEL 7.9 with lsf scheduler.

When I run command $ nextflow distiller.nf -params-file ./test/test_project.yml -profile cluster

hit an error: Error executing process > 'download_truncate_chunk_fastqs (library:MATalpha_R1 run:lane2)'

Caused by: Failed to submit process to grid scheduler for execution

Command executed:

bsub

Command exit status: 255

Command output: MEMLIMIT value <0> is not valid. Limit must be a positive integer. You can use the following units for the limit: KB (or K), MB (or M), GB (or G), TB (or T), PB (or P), EB (or E), ZB (or Z). Job not submitted.

at the beggining of /mytest/work/46/3cb7ede31d929c5f37a8421acf1932.command.run: #!/bin/bash #BSUB -o /mytest/work/46/3cb7ede31d929c5f37a8421acf1932/.command.log #BSUB -q short #BSUB -n 4 #BSUB -R "span[hosts=1]" #BSUB -W 02:00 #BSUB -M 0 #BSUB -R "select[mem>=2] rusage[mem=2]" #BSUB -J nf-download_truncate_chunk_fastqs_(library_MATalpha_R1_run_lane2)

memory =0 is not acceptable on our cluster. Apparently, -M 0 and -R should be adjusted to fit our system. But I have no idea how these value get passed. in configs/cluster.config I see the following defined for function download_truncate_chunk_fastqs. it has memory = '2 GB', somewhere else I should look for?

withName: download_truncate_chunk_fastqs { maxForks = 10 cpus = 4 memory = '2 GB' queue = 'short' time = '2h' }

Thank you very much.

ryao-mdanderson avatar Jun 16 '22 22:06 ryao-mdanderson