failed job run_mmseqs

[jiyunli]

Hi phil@philarevalo I meet a problem when running "bash snakefile.sh" in flexible_genome_sweeps. the following is the error. Waiting at most 5 seconds for missing files. MissingOutputException in line 88 of /disk1/cau/cvmljy/pop/PopCOGenT-master/src/flexible_genome_sweeps/Snakefile: Missing files after 5 seconds: proc/sulfolobus/clusters/clu.0 This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Removing output files of failed job run_mmseqs since they might be corrupted: proc/sulfolobus/clusters/DB.0 Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message

[philarevalo]

Hi Yun,

I was unable to recreate this on my own system. I think one possibility might be that you may be running a newer version of mmseqs2 than we tested with. Can you install the following version and see if that fixes the problem? If so, we'll update the documentation.

https://github.com/soedinglab/MMseqs2/releases/tag/vNatBiotech

If that doesn't work, could you answer and try the following?

How many threads are you using and what kind of node are you running this on (operating system, number of cores, memory)?

Can you see if it works if you modify snakefile.sh to the following:

#!/bin/bash
source activate PopCOGenT
snakemake --latency-wait 600

[jiyunli]

Thanks Phil @philarevalo There are some parameter: 10 threads were applied. operating system: CentOS（using computer cluster server） number of cores: 12 (cat /proc/cpuinfo |grep "processor"|sort -u|wc -l)

It doesn't work when using the mmseqs2 from website provided. It also doesn't work when add "--latency-wait 600" to snake file.sh. MissingOutputException in line 88 of /disk1/cau/cvmlll/pop/PopCOGenT-master/src/flexible_genome_sweeps/Snakefile: Missing files after 600 seconds: proc/sulfolobus/clusters/clu.0 This might be due to filesystem latency. If that is the case, consider to increase the wait time with --latency-wait. Removing output files of failed job run_mmseqs since they might be corrupted: proc/sulfolobus/clusters/DB.0 Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message

[elsherbini]

Hi Yun,

Can you confim that you have a file called proc/sulfolobus/0.concatenated_orfs.faa , and post the output of the following commands?

head proc/sulfolobus/0.concatenated_orfs.faa
grep -c ">" proc/sulfolobus/0.concatenated_orfs.faa

[jiyunli]

"grep -c ">" proc/sulfolobus/0.concatenated_orfs.faa" is print "35906" and there have a file called proc/sulfolobus/0.concatenated_orfs.faa. "head proc/sulfolobus/0.concatenated_orfs.faa" as following:

M1643_contig_0_1 # 466 # 762 # 1 # ID=1_1;partial=00;start_type=ATG;rbs_motif=None;rbs_spacer=None;gc_cont=0.269 MIIRRMFIKSTLAVSFSISYLLYIIAKISFGSGKLAFIIALLQLYPYFLIDYLPTVQVVS LFFQSLLLLLFAIFILLITYMLFLFYKLYKMVKTSILK* M1643_contig_0_2 # 843 # 2024 # 1 # ID=1_2;partial=00;start_type=GTG;rbs_motif=None;rbs_spacer=None;gc_cont=0.365 MIRETLKGGKGEVIKNPKVFIDPLSVFKEIPFREDILRDAAIAIRYFVKNEVKFSNLFLG LTGTGKTFVSKYIYNEIEEVKNEDEEYKDVKQAYVNCREVGGTPQAVLSSLAGKLTGFSV PKHGINLGEYIDKIKNGTRNIKAIIYLDEVDTLVKRRGGDIVLYQLLRSDANISVIMISN DINVRDYMEPRVLSSLGPSVIFKPYDAEQLKFILSKYAEYGLIKGTYDGDILGYIAAISA KEHGDARKAVNLLFRAAQLASGGGMIKKEHVDKAIVDYEQERLIEAVKALPFHYKLALRS LIESEDVMSAHKMYTDLCNKFKQKPLSYRRFSDIISELDMFGIVKIRIVNRGRAGGVKKY

[asierzaragoza]

Just in case someone else is having this issue and is looking for a quick solution, I managed to get the pipeline running by editing the snakemake script. Look for this section of the code in /PopCOGenT/src/flexible_genome_sweeps/Snakefile :

rule run_mmseqs:
    input:
        "proc/{organism}/%s.concatenated_orfs.faa" % config['Group_to_cluster']
    output:
        mmseqs_db = "proc/{organism}/clusters/DB.%s" % config['Group_to_cluster'],
        clusters = "proc/{organism}/clusters/clu.%s" % config['Group_to_cluster']
    threads:
        int(config['num_threads'])
    shell:
        """
        mkdir -p ./tmp/tmp_{wildcards.organism}/mmseqs;
        rm -rf ./tmp/tmp_{wildcards.organism}/mmseqs/*;
        {config[mmseqs_path]} createdb {input} {output.mmseqs_db};
        {config[mmseqs_path]} cluster {output.mmseqs_db} {output.clusters} ./tmp/tmp_{wildcards.organism}/mmseqs --min-seq-id 0.50 --max-seqs 100000 -c 0.8;
        """

rule clusters_to_tsv:
    input:
        mmseqs_db = "proc/{organism}/clusters/DB.%s" % config['Group_to_cluster'],
        clusters = "proc/{organism}/clusters/clu.%s" % config['Group_to_cluster']
    output:
        clusters = "proc/{organism}/clusters/clusters.%s.tsv" % config['Group_to_cluster']
    shell:
        "{config[mmseqs_path]} createtsv {input.mmseqs_db} {input.mmseqs_db} {input.clusters} {output}"

Remove the 'clusters' variable from the output section in run_mmseqs and the input section in clusters_to_tsv, and replace their respective calls in the shell section with the path to the mmseqs cluster output. Here is an example:

rule run_mmseqs:
    input:
        "proc/{organism}/%s.concatenated_orfs.faa" % config['Group_to_cluster']
    output:
        mmseqs_db = "proc/{organism}/clusters/DB.%s" % config['Group_to_cluster']
    threads:
        int(config['num_threads'])
    shell:
        """
        mkdir -p ./tmp/tmp_{wildcards.organism}/mmseqs;
        rm -rf ./tmp/tmp_{wildcards.organism}/mmseqs/*;
        {config[mmseqs_path]} createdb {input} {output.mmseqs_db};
        {config[mmseqs_path]} cluster {output.mmseqs_db} ./proc/test_run/clusters/clu.0 ./tmp/tmp_{wildcards.organism}/mmseqs --min-seq-id 0.50 --max-seqs 100000 -c 0.8;
        """

rule clusters_to_tsv:
    input:
        mmseqs_db = "proc/{organism}/clusters/DB.%s" % config['Group_to_cluster']
    output:
        clusters = "proc/{organism}/clusters/clusters.%s.tsv" % config['Group_to_cluster']
    shell:
        "{config[mmseqs_path]} createtsv {input.mmseqs_db} {input.mmseqs_db} ./proc/test_run/clusters/clu.0 {output}"

Looks like the error is not due to latency issues, but is caused by the snakemake script waiting for a mmseqs output that does not exist. (In my case, I never got the single clu.0 file that the program expects, but a series of files named clu.0.0, clu.0.1, etc).