distiller-nf
distiller-nf copied to clipboard
how memory limit gets specified in cluster using
Dear distiller-nf support,
This is a beginner question. I am using the test example on an HPC cluster RHEL 7.9 with lsf scheduler.
When I run command $ nextflow distiller.nf -params-file ./test/test_project.yml -profile cluster
hit an error: Error executing process > 'download_truncate_chunk_fastqs (library:MATalpha_R1 run:lane2)'
Caused by: Failed to submit process to grid scheduler for execution
Command executed:
bsub
Command exit status: 255
Command output: MEMLIMIT value <0> is not valid. Limit must be a positive integer. You can use the following units for the limit: KB (or K), MB (or M), GB (or G), TB (or T), PB (or P), EB (or E), ZB (or Z). Job not submitted.
at the beggining of /mytest/work/46/3cb7ede31d929c5f37a8421acf1932.command.run: #!/bin/bash #BSUB -o /mytest/work/46/3cb7ede31d929c5f37a8421acf1932/.command.log #BSUB -q short #BSUB -n 4 #BSUB -R "span[hosts=1]" #BSUB -W 02:00 #BSUB -M 0 #BSUB -R "select[mem>=2] rusage[mem=2]" #BSUB -J nf-download_truncate_chunk_fastqs_(library_MATalpha_R1_run_lane2)
memory =0 is not acceptable on our cluster. Apparently, -M 0 and -R should be adjusted to fit our system. But I have no idea how these value get passed. in configs/cluster.config I see the following defined for function download_truncate_chunk_fastqs. it has memory = '2 GB', somewhere else I should look for?
withName: download_truncate_chunk_fastqs { maxForks = 10 cpus = 4 memory = '2 GB' queue = 'short' time = '2h' }
Thank you very much.