metAMOS icon indicating copy to clipboard operation
metAMOS copied to clipboard

Published 20 hours ago • verified publisher icon marbl

rm '/path/Preprocess/out/all.seq.mates': No such file or directory

Open alneberg opened this issue 10 years ago • 6 comments

Hi again,

I'm trying to run the iMetAMOS workflow with the following commands:

initPipeline -f -1 ../../reads/120920-bin57_120920_R1.fasta -2 ../../reads/120920-bin57_120920_R2.fasta -d 120920_57 -W iMetAMOS -i 0:800
runPipeline -d 120920_57 -p 8

But I recieve the message:

Starting Task = preprocess.PREPROCESS
Oops, MetAMOS finished with errors! see text in red above for details.

and the log in 120920_57/Logs/PREPROCESS.log says:

rm: cannot remove `/<path>/120920_57/Preprocess/out/all.seq.mates': No such file or directory

The path for Preprocess/out/ is correct but there is no such file, only:

$ ls 120920_57/Preprocess/out/
120920-bin57_120920_R1.fasta  120920-bin57_120920_R1.fasta.qual  120920-bin57_120920_R2.fasta  120920-bin57_120920_R2.fasta.qual

which are all empty.

At the beginning of the run, metAMOS prints:

Error: cannot find BLAST DB directory, expected it in /pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/DB/. Disabling blastdb dependent programs
**no blast DB directory available, disabling steps requiring BLAST DB
[Steps to be skipped]:  set(['FunctionalAnnotation', 'FindScaffoldORFS', 'FindRepeats'])
Starting metAMOS pipeline
Error: cannot find BLAST DB directory, expected it in /pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/DB/. Disabling blastdb dependent programs
Warning: BLASR is not found, some functionality will not be available
Warning: Newbler is not found, some functionality will not be available
Warning: MetaGeneMark is not found, some functionality will not be available
Warning: SignalP+ is not found, some functionality will not be available
Warning: metaphylerClassify is not found, some functionality will not be available
Warning: PhyloSift was not found, will not be available

Warning: FreeBayes is not found, some functionality will not be available

Is it any of the missing programs that causes the problem?

Thanks in advance, Johannes

alneberg avatar Jul 31 '14 11:07 alneberg

Hi,

I think the issue is that the minimum insert size is set to 0, which is causing the error. If you set the insert size to at least 1 it should work.

skoren avatar Aug 01 '14 00:08 skoren

Thank you for your suggestion, but unfortunately I get the same error when running with 1, 800 instead of 0,800.

alneberg avatar Aug 01 '14 09:08 alneberg

Hi,

The likeliest cause is a malformed input file. Could you share your input fasta sequences (or a subset of them required to reproduce the error) along with the output from:

initPipeline -f -1 ../../reads/120920-bin57_120920_R1.fasta -2 ../../reads/120920-bin57_120920_R2.fasta -d 120920_57 -W iMetAMOS -i 0:800
runPipeline -d 120920_57 -p 8 -v

skoren avatar Aug 01 '14 13:08 skoren

Sure!

I subsetted the read files substantially, leaving:

$ cat 120920_R1.fasta 
>HISEQ:99:D2GAAACXX:3:2310:2723:38028
AACAAAAAACAACTCTGTCTCCTCAAGATTTGTGCTGTTGCAATCTGACAGCTTGCCTGG
CAGTGCCGGACCTTGAAAGGTCTTCTTTCAGATCGGAAGAG
>HISEQ:99:D2GAAACXX:3:2314:5011:91594
AACAAAAAACCCAAAGACCCACAGCAGAAATAAATATATTGTTTTCATTTGGGGAGCAAG
CGTCTGTTCAGCCTCTTGAAGAATTTCAGTTGGCAAGATAA
>HISEQ:99:D2GAAACXX:3:2210:15032:37405
AACAAAAAACCTGGCAACCTGAATGACATCTGCTGTTTTACCAAAATAGTTGACATCAAA
CTCGGCTAAAACTAGGTTCACGAACAAGGCATTGAACATAG
>HISEQ:99:D2GAAACXX:3:2211:10647:83679
AACAAAAAAGAAATAAATAAAATTCTTGAGTCTGTCAGCCTGCAAGGTCTTACCTGTGTT
CTAACAATGGTTTACTGGAAGGAAAATAAAATTAAGTGCGA
>HISEQ:99:D2GAAACXX:3:2216:1667:79224
AACAAAAAAGAAATAAATAAAATTCTTGAGTCTGTCAGCCTGCAAGGTCTTACCTGTGTT
CTAACAATGGTTTACTGGAAGGAAAATAAAATTAAGTGCGA
>HISEQ:99:D2GAAACXX:3:2216:19582:39059
AACAAAAAAGACATCGTGCATTCCCATTAACCCAAGAAGGCTCATGGAGCAAAGAGGTGC
AGCATAACTTCGCATTTCAACTGACCCAGGCTCAGTCGCGA
>HISEQ:99:D2GAAACXX:3:1211:9410:26283
AACAAAAAAGAGCTACACCACATTGAATCTATTTTAAGAGCCATGAACGCAGAAAAACTT
TTAGACCAAGGCATCACACTAACAGACGCCACCCGTGTAGA

$ cat 120920_R2.fasta
>HISEQ:99:D2GAAACXX:3:2310:2723:38028
GAAAGAAGACCTTTCAAGGTCCGGCACTGCCAGGCAAGCTGTCAGATTGCAACAGCACAA
ATCTTGAGGAGACAGAGTTGTTTTTTGTTAGATCGGAAGAG
>HISEQ:99:D2GAAACXX:3:2314:5011:91594
CAAACATGGTTCTTAGAAAAAGGAGCGTTTATCTTGCCAACTGAAATTCTTCAAGAGGCT
GAACAGACGCTTGCTCCCCAAATGAAAACAATATATTTATT
>HISEQ:99:D2GAAACXX:3:2210:15032:37405
CAAGGCAGAGCCCTTTGAACAAGAGATATTATCGGCACATCGCCTTGATCGTGATGGTTT
CTGCTATGTTCAATGCCTTGTTCGTGAACCTAGTTTTAGCC
>HISEQ:99:D2GAAACXX:3:2211:10647:83679
TCGCACTTAATTTTATTTTCCTTCCAGTAAACCATTGTTAGAACACAGGTAAGACCTTGC
AGGCTGACAGACTCAAGAATTTTATTTATTTCTTTTTTGTT
>HISEQ:99:D2GAAACXX:3:2216:1667:79224
TGGTTAGCCTTAAGCGGCTAAAGCGTAGTTGTCGTTTTCTGCAACTAATTTTTTTGTAGT
AAGTTTTACAAGAGTTACCACAATCTTGGCATGCGCTTTGG
>HISEQ:99:D2GAAACXX:3:2216:19582:39059
ATCGCATCATAGGAACTGCTTGAGACAGATCATGCCGAATCTCTTCTAGGACTCGCGACT
GAGCCTGGGTCAGTTGAAATGCGAAGTTATGCTGCACCTCT
>HISEQ:99:D2GAAACXX:3:1211:9410:26283
CCTGCATTTAGTTCGCCGCGAAGGTCTACACGGGTGGCGTCTGTTAGTGTGATGCCTTGG
TCTAAAAGTTTTTCTGCGTTCATGGCTCTTAAAATAGATTC

Running:

$ initPipeline -f -1 120920_R1.fasta -2 120920_R2.fasta -d metamos_outdir -W iMetAMOS -i 1:800
Error: cannot find BLAST DB directory, expected it in /pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/DB/. Disabling blastdb dependent programs
Project dir /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir successfully created!
Use runPipeline.py to start Pipeline

$ runPipeline -d metamos_outdir -p 8 -v
....
blastdb warnings + citations
.....
Traceback (most recent call last):
  File "/proj/b2010008/bin/runPipeline", line 984, in <module>
    verbose = 1)
  File "/pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/ruffus/task.py", line 2965, in pipeline_run
    raise job_errors
RethrownJobError: 



    Exception #1
      'exceptions.AttributeError('str' object has no attribute 'append')' raised in ...
       Task = def preprocess.Preprocess(...):
       Job  = [[120920_R1.fasta, 120920_R2.fasta] -> preprocess.success]

    Traceback (most recent call last):
      File "/pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/ruffus/task.py", line 625, in run_pooled_job_without_exceptions
        return_value =  job_wrapper(param, user_defined_work_func, register_cleanup, touch_files_only)
      File "/pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/ruffus/task.py", line 491, in job_wrapper_io_files
        ret_val = user_defined_work_func(*param)
  File "/pica/v1/b2010008/src/metAMOS-1.5rc3/src/preprocess.py", line 470, in Preprocess
    qs2.append(line.rstrip())
AttributeError: 'str' object has no attribute 'append'


 Starting Task = preprocess.PREPROCESS
*** metAMOS running command: java -cp /pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/java:.:. outputDefaultQuality /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir/Preprocess/in/120920_R1.fasta > /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir/Preprocess/in/120920_R1.fasta.qual

*** metAMOS running command: java -cp /pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/java:.:. outputDefaultQuality /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir/Preprocess/in/120920_R2.fasta > /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir/Preprocess/in/120920_R2.fasta.qual

 *** metAMOS running command: rm /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir/Preprocess/out/all.seq.mates

Oops, MetAMOS finished with errors! see text in red above for details.

Hope this helps! Cheers!

alneberg avatar Aug 01 '14 14:08 alneberg

hi Johannes,

Thanks for sending the detailed output. Upon closer inspection, this indeed appears to be a bug for non-interleaved, paired, fasta reads when default metamos read filtering is used. We are currently prepping a patch to fix this (will be released under v1.5) and will let you know as soon as it becomes available. In the meantime, either interleaving your fasta files or enabling eautils (-t eautils) read filtering should bypass this error.

-Todd

On Fri, Aug 1, 2014 at 10:06 AM, Johannes Alneberg <[email protected]

wrote:

Sure!

I subsetted the read files substantially, leaving:

$ cat 120920_R1.fasta

HISEQ:99:D2GAAACXX:3:2310:2723:38028 AACAAAAAACAACTCTGTCTCCTCAAGATTTGTGCTGTTGCAATCTGACAGCTTGCCTGG CAGTGCCGGACCTTGAAAGGTCTTCTTTCAGATCGGAAGAG HISEQ:99:D2GAAACXX:3:2314:5011:91594 AACAAAAAACCCAAAGACCCACAGCAGAAATAAATATATTGTTTTCATTTGGGGAGCAAG CGTCTGTTCAGCCTCTTGAAGAATTTCAGTTGGCAAGATAA HISEQ:99:D2GAAACXX:3:2210:15032:37405 AACAAAAAACCTGGCAACCTGAATGACATCTGCTGTTTTACCAAAATAGTTGACATCAAA CTCGGCTAAAACTAGGTTCACGAACAAGGCATTGAACATAG HISEQ:99:D2GAAACXX:3:2211:10647:83679 AACAAAAAAGAAATAAATAAAATTCTTGAGTCTGTCAGCCTGCAAGGTCTTACCTGTGTT CTAACAATGGTTTACTGGAAGGAAAATAAAATTAAGTGCGA HISEQ:99:D2GAAACXX:3:2216:1667:79224 AACAAAAAAGAAATAAATAAAATTCTTGAGTCTGTCAGCCTGCAAGGTCTTACCTGTGTT CTAACAATGGTTTACTGGAAGGAAAATAAAATTAAGTGCGA HISEQ:99:D2GAAACXX:3:2216:19582:39059 AACAAAAAAGACATCGTGCATTCCCATTAACCCAAGAAGGCTCATGGAGCAAAGAGGTGC AGCATAACTTCGCATTTCAACTGACCCAGGCTCAGTCGCGA HISEQ:99:D2GAAACXX:3:1211:9410:26283 AACAAAAAAGAGCTACACCACATTGAATCTATTTTAAGAGCCATGAACGCAGAAAAACTT TTAGACCAAGGCATCACACTAACAGACGCCACCCGTGTAGA

$ cat 120920_R2.fasta

HISEQ:99:D2GAAACXX:3:2310:2723:38028 GAAAGAAGACCTTTCAAGGTCCGGCACTGCCAGGCAAGCTGTCAGATTGCAACAGCACAA ATCTTGAGGAGACAGAGTTGTTTTTTGTTAGATCGGAAGAG HISEQ:99:D2GAAACXX:3:2314:5011:91594 CAAACATGGTTCTTAGAAAAAGGAGCGTTTATCTTGCCAACTGAAATTCTTCAAGAGGCT GAACAGACGCTTGCTCCCCAAATGAAAACAATATATTTATT HISEQ:99:D2GAAACXX:3:2210:15032:37405 CAAGGCAGAGCCCTTTGAACAAGAGATATTATCGGCACATCGCCTTGATCGTGATGGTTT CTGCTATGTTCAATGCCTTGTTCGTGAACCTAGTTTTAGCC HISEQ:99:D2GAAACXX:3:2211:10647:83679 TCGCACTTAATTTTATTTTCCTTCCAGTAAACCATTGTTAGAACACAGGTAAGACCTTGC AGGCTGACAGACTCAAGAATTTTATTTATTTCTTTTTTGTT HISEQ:99:D2GAAACXX:3:2216:1667:79224 TGGTTAGCCTTAAGCGGCTAAAGCGTAGTTGTCGTTTTCTGCAACTAATTTTTTTGTAGT AAGTTTTACAAGAGTTACCACAATCTTGGCATGCGCTTTGG HISEQ:99:D2GAAACXX:3:2216:19582:39059 ATCGCATCATAGGAACTGCTTGAGACAGATCATGCCGAATCTCTTCTAGGACTCGCGACT GAGCCTGGGTCAGTTGAAATGCGAAGTTATGCTGCACCTCT HISEQ:99:D2GAAACXX:3:1211:9410:26283 CCTGCATTTAGTTCGCCGCGAAGGTCTACACGGGTGGCGTCTGTTAGTGTGATGCCTTGG TCTAAAAGTTTTTCTGCGTTCATGGCTCTTAAAATAGATTC

Running:

$ initPipeline -f -1 120920_R1.fasta -2 120920_R2.fasta -d metamos_outdir -W iMetAMOS -i 1:800 Error: cannot find BLAST DB directory, expected it in /pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/DB/. Disabling blastdb dependent programs Project dir /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir successfully created! Use runPipeline.py to start Pipeline

$ runPipeline -d metamos_outdir -p 8 -v .... blastdb warnings + citations ..... Traceback (most recent call last): File "/proj/b2010008/bin/runPipeline", line 984, in verbose = 1) File "/pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/ruffus/task.py", line 2965, in pipeline_run raise job_errors RethrownJobError:

Exception #1
  'exceptions.AttributeError('str' object has no attribute 'append')' raised in ...
   Task = def preprocess.Preprocess(...):
   Job  = [[120920_R1.fasta, 120920_R2.fasta] -> preprocess.success]

Traceback (most recent call last):
  File "/pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/ruffus/task.py", line 625, in run_pooled_job_without_exceptions
    return_value =  job_wrapper(param, user_defined_work_func, register_cleanup, touch_files_only)
  File "/pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/ruffus/task.py", line 491, in job_wrapper_io_files
    ret_val = user_defined_work_func(*param)

File "/pica/v1/b2010008/src/metAMOS-1.5rc3/src/preprocess.py", line 470, in Preprocess qs2.append(line.rstrip()) AttributeError: 'str' object has no attribute 'append'

Starting Task = preprocess.PREPROCESS *** metAMOS running command: java -cp /pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/java:.:. outputDefaultQuality /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir/Preprocess/in/120920_R1.fasta > /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir/Preprocess/in/120920_R1.fasta.qual

*** metAMOS running command: java -cp /pica/v1/b2010008/src/metAMOS-1.5rc3/Utilities/java:.:. outputDefaultQuality /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir/Preprocess/in/120920_R2.fasta > /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir/Preprocess/in/120920_R2.fasta.qual

*** metAMOS running command: rm /gulo/proj_nobackup/b2010008/projects/luisa-baltic/2012_sg/sample_cutup_concocts/reads_per_good_cluster/assemblies/imetamos/small_files/metamos_outdir/Preprocess/out/all.seq.mates

Oops, MetAMOS finished with errors! see text in red above for details.

Hope this helps! Cheers!

— Reply to this email directly or view it on GitHub https://github.com/marbl/metAMOS/issues/149#issuecomment-50887814.

treangen avatar Aug 06 '14 11:08 treangen

Hi @treangen,

thank you for your reply. I will try one of your suggested solutions and get back if none work.

Cheers, Johannes

alneberg avatar Aug 06 '14 12:08 alneberg