Warning: unable to close filehandle properly: No space left on device during global destruction.

[johnsolk]

Not sure if this is a dammit problem, seems like a disk space issue. parallel during lastall seemed unhappy with long records:

# dammit
## a tool for easy de novo transcriptome annotation

by Camille Scott

**v1.1**, 2018

## submodule: annotate
### Database Check
#### Info
* Database Directory: /pylon5/bi5fpmp/ljcohen/dammit
* Doit Database: /pylon5/bi5fpmp/ljcohen/dammit/databases.doit.db


*All database tasks up-to-date.*

### Annotation
#### Info
* Doit Database: /local/4697162/F_notti/F_notti.dammit/annotate.doit.db
* Input Transcriptome: /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta


Some tasks out of date!


Out-of-date tasks:
* BUSCO-eukaryota
* TransDecoder.LongOrfs
* TransDecoder.Predict
* annotate:fasta
* cmscan:Rfam
* gff3:/local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.Fundulus_heteroclitus.Fundulus_heteroclitus-3.0.2.pep.all.fa.crbl.csv
* gff3:/local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.kfish2rae5g.pub.aa.crbl.csv
* gff3:/local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.protein.fa.crbl.csv
* gff3:OrthoDB
* gff3:Pfam-A
* gff3:Rfam
* gff3:merge-all
* gff3:sprot
* hmmscan:Pfam-A
* hmmscan:Pfam-A:remap
* lastal:OrthoDB
* lastal:best-hits:OrthoDB
* lastal:best-hits:sprot
* lastal:sprot
* rename-transcriptome
* transcriptome-stats
* user-database:Fundulus_heteroclitus.Fundulus_heteroclitus-3.0.2.pep.all.fa-shmlast-fit_and_filter_crbl_hits
* user-database:Fundulus_heteroclitus.Fundulus_heteroclitus-3.0.2.pep.all.fa-shmlast-lastal:.F_notti.trinity_out.Trinity.fasta.pep.x.Fundulus_heteroclitus.Fundulus_heteroclitus-3.0.2.pep.all.fa.maf
* user-database:Fundulus_heteroclitus.Fundulus_heteroclitus-3.0.2.pep.all.fa-shmlast-lastal:.Fundulus_heteroclitus.Fundulus_heteroclitus-3.0.2.pep.all.fa.x.F_notti.trinity_out.Trinity.fasta.pep.maf
* user-database:Fundulus_heteroclitus.Fundulus_heteroclitus-3.0.2.pep.all.fa-shmlast-lastdb:.Fundulus_heteroclitus.Fundulus_heteroclitus-3.0.2.pep.all.fa
* user-database:Fundulus_heteroclitus.Fundulus_heteroclitus-3.0.2.pep.all.fa-shmlast-rename:/local/4697162/F_notti/Fhet_reference_genome/ensembl/Fundulus_heteroclitus.Fundulus_heteroclitus-3.0.2.pep.all.fa
* user-database:kfish2rae5g.pub.aa-shmlast-fit_and_filter_crbl_hits
* user-database:kfish2rae5g.pub.aa-shmlast-lastal:.F_notti.trinity_out.Trinity.fasta.pep.x.kfish2rae5g.pub.aa.maf
* user-database:kfish2rae5g.pub.aa-shmlast-lastal:.kfish2rae5g.pub.aa.x.F_notti.trinity_out.Trinity.fasta.pep.maf
* user-database:kfish2rae5g.pub.aa-shmlast-lastdb:.kfish2rae5g.pub.aa
* user-database:kfish2rae5g.pub.aa-shmlast-rename:/local/4697162/F_notti/Fhet_reference_genome/evigene/kfish2rae5g.pub.aa
* user-database:protein.fa-shmlast-fit_and_filter_crbl_hits
* user-database:protein.fa-shmlast-lastal:.F_notti.trinity_out.Trinity.fasta.pep.x.protein.fa.maf
* user-database:protein.fa-shmlast-lastal:.protein.fa.x.F_notti.trinity_out.Trinity.fasta.pep.maf
* user-database:protein.fa-shmlast-lastdb:.F_notti.trinity_out.Trinity.fasta.pep
* user-database:protein.fa-shmlast-lastdb:.protein.fa
* user-database:protein.fa-shmlast-rename:/local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta
* user-database:protein.fa-shmlast-rename:/local/4697162/F_notti/Fhet_reference_genome/ncbi/protein.fa
* user-database:protein.fa-shmlast-translate:.F_notti.trinity_out.Trinity.fasta

#### Run Tasks
- [ ] F_notti.trinity_out.Trinity.fasta: 
    * Python: function get_rename_transcriptome_task.fix
- [ ] transcriptome_stats:F_notti.trinity_out.Trinity.fasta: 
    * Python: function get_transcriptome_stats_task.cmd
- [ ] busco:F_notti.trinity_out.Trinity.fasta-eukaryota_odb9: 
    * Cmd: `python3 /pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/run_BUSCO.py -i /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta -f -o F_notti.trinity_out.Trinity.fasta.eukaryota.busco.results -l /pylon5/bi5fpmp/ljcohen/dammit/busco2db/eukaryota_odb9 -m tran -c 14`
- [ ] TransDecoder.LongOrfs:F_notti.trinity_out.Trinity.fasta: 
    * Cmd: `/pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/TransDecoder.LongOrfs -t /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta -m 80`
- [ ] hmmscan:longest_orfs.pep.x.Pfam-A.hmm: 
    * Cmd: `cat /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.transdecoder_dir/longest_orfs.pep | /pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/parallel --block `expr $(wc -c /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.transdecoder_dir/longest_orfs.pep | awk '{print $1}') / 14` --round-robin --pipe --recstart '>' --gnu -j 14 /pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/hmmscan --cpu 1 --domtblout /dev/stdout -E 1e-05 -o /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.transdecoder_dir/longest_orfs.pep.x.pfam.tbl.hmmscan.out /pylon5/bi5fpmp/ljcohen/dammit/Pfam-A.hmm /dev/stdin > /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.transdecoder_dir/longest_orfs.pep.x.pfam.tbl`
- [ ] remap_hmmer:longest_orfs.pep.x.pfam.tbl: 
    * Python: function get_remap_hmmer_task.cmd
- [ ] hmmscan-gff3:F_notti.trinity_out.Trinity.fasta.x.pfam-A.gff3: 
    * Cmd: `rm -f /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.pfam-A.gff3`
    * Python: function get_hmmscan_gff3_task.cmd
- [ ] TransDecoder.Predict:F_notti.trinity_out.Trinity.fasta: 
    * Cmd: `/pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/TransDecoder.Predict -t /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta --retain_pfam_hits /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.transdecoder_dir/longest_orfs.pep.x.pfam.tbl`
- [ ] cmscan:F_notti.trinity_out.Trinity.fasta.x.Rfam.cm: 
    * Cmd: `cat /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta | /pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/parallel --block `expr $(wc -c /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta | awk '{print $1}') / 14` --round-robin --pipe --recstart '>' --gnu -j 14 /pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/cmscan --cpu 1 --rfam --nohmmonly -E 1e-05 --tblout /dev/stdout -o /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.rfam.tbl.cmscan.out /pylon5/bi5fpmp/ljcohen/dammit/Rfam.cm /dev/stdin > /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.rfam.tbl`
- [ ] cmscan-gff3:F_notti.trinity_out.Trinity.fasta.x.rfam.gff3: 
    * Cmd: `rm -f /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.rfam.gff3`
    * Python: function get_cmscan_gff3_task.cmd
- [ ] lastal:/local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.OrthoDB.maf: 
    * Python: function setup_profiler.add_profile_actions.start_profiling
    * Cmd: `cat /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta | /pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/parallel --round-robin --pipe -L 2 -N 10000 --gnu -j 14 -a /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta /pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/lastal -F15 -D100000.0 /pylon5/bi5fpmp/ljcohen/dammit/aa_seq_euk.fasta > /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.OrthoDB.maf`
    * Python: function setup_profiler.add_profile_actions.stop_profiling
TaskError - taskid:lastal:/local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.OrthoDB.maf
Command error: 'cat /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta | /pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/parallel --round-robin --pipe -L 2 -N 10000 --gnu -j 14 -a /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta /pylon5/bi5fpmp/ljcohen/miniconda3/envs/dammit_master/bin/lastal -F15 -D100000.0 /pylon5/bi5fpmp/ljcohen/dammit/aa_seq_euk.fasta > /local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.OrthoDB.maf' returned 255

########################################
TaskError - taskid:lastal:/local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.OrthoDB.maf
lastal:/local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.OrthoDB.maf <stderr>:
parallel: Warning: A record was longer than 1048576. Increasing to --blocksize 1363150.
parallel: Warning: A record was longer than 1363150. Increasing to --blocksize 1772096.
parallel: Warning: A record was longer than 1772096. Increasing to --blocksize 2303726.
parallel: Warning: A record was longer than 2303726. Increasing to --blocksize 2994845.
parallel: Warning: A record was longer than 2994845. Increasing to --blocksize 3893300.
parallel: Warning: A record was longer than 3893300. Increasing to --blocksize 5061291.
parallel: Warning: A record was longer than 5061291. Increasing to --blocksize 6579680.
parallel: Warning: A record was longer than 6579680. Increasing to --blocksize 8553585.
parallel: Warning: A record was longer than 8553585. Increasing to --blocksize 11119662.
parallel: Warning: A record was longer than 11119662. Increasing to --blocksize 14455562.
parallel: Warning: A record was longer than 14455562. Increasing to --blocksize 18792232.
parallel: Warning: A record was longer than 18792232. Increasing to --blocksize 24429903.
parallel: Error: Output is incomplete.
parallel: Error: Cannot append to buffer file in /tmp.
parallel: Error: Is the disk full?
parallel: Error: Change $TMPDIR with --tmpdir or use --compress.
Warning: unable to close filehandle properly: No space left on device during global destruction.

lastal:/local/4697162/F_notti/F_notti.dammit/F_notti.trinity_out.Trinity.fasta.x.OrthoDB.maf <stdout>:

[camillescott]

Interesting. The --blocksize issues are just warnings, parallel will adjust on its own. The other issue though, it looks like it's running out of disk space for its buffer. Is this on a Amazon instance?

[johnsolk]

No, hpc - PSC Bridges. I have to write to temp $LOCAL node disk space then copy output at the end to avoid hammering storage disk. I ran it again and it seems to have worked. (yay!? although, my .o files are messed up.) So, maybe a node disk space issue?

[schwarl]

Hi! I am having a similar issue, here is my err file:

dammit

a tool for easy de novo transcriptome annotation

by Camille Scott

v1.0rc2, 2018

submodule: annotate

Database Check

Info

• Database Directory: /home/schwarl/.dammit/databases
• Doit Database: /home/schwarl/.dammit/databases/databases.doit.db

All database tasks up-to-date.

Annotation

Info

• Doit Database: /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/annotate.doit.db
• Input Transcriptome: /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta

Some tasks out of date!

Up-to-date tasks:

• TransDecoder.LongOrfs
• TransDecoder.Predict
• cmscan:Rfam
• gff3:Pfam-A
• gff3:Rfam
• hmmscan:Pfam-A
• hmmscan:Pfam-A:remap
• rename-transcriptome
• transcriptome-stats

Out-of-date tasks:

• BUSCO-metazoa
• annotate:fasta
• gff3:OrthoDB
• gff3:merge-all
• lastal:OrthoDB
• lastal:best-hits:OrthoDB

Run Tasks

• [x] edulis.Trinity.fasta:
  • Python: function get_rename_transcriptome_task.fix
• [x] transcriptome_stats:edulis.Trinity.fasta:
  • Python: function get_transcriptome_stats_task.cmd
• [ ] busco:edulis.Trinity.fasta-metazoa_odb9:
  • Cmd: python3 /home/schwarl/.conda/envs/py3.dammit/bin/run_BUSCO.py -i /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta -f -o edulis.Trinity.fasta.metazoa.busco.results -l /home/schwarl/.dammit/databases/busco2db/metazoa_odb9 -m tran -c 20
• [x] TransDecoder.LongOrfs:edulis.Trinity.fasta:
  • Cmd: /home/schwarl/.conda/envs/py3.dammit/bin/TransDecoder.LongOrfs -t /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta -m 80
• [x] hmmscan:longest_orfs.pep.x.Pfam-A.hmm:
  • Cmd: cat /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.transdecoder_dir/longest_orfs.pep | /home/schwarl/.conda/envs/py3.dammit/bin/parallel --round-robin --pipe -L 2 -N 10000 --gnu -j 20 -a /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.transdecoder_dir/longest_orfs.pep /home/schwarl/.conda/envs/py3.dammit/bin/hmmscan --cpu 1 --domtblout /dev/stdout -E 1e-05 -o /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.transdecoder_dir/longest_orfs.pep.x.pfam.tbl.hmmscan.out /home/schwarl/.dammit/databases/Pfam-A.hmm /dev/stdin > /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.transdecoder_dir/longest_orfs.pep.x.pfam.tbl
• [x] remap_hmmer:longest_orfs.pep.x.pfam.tbl:
  • Python: function get_remap_hmmer_task.cmd
• [x] hmmscan-gff3:edulis.Trinity.fasta.x.pfam-A.gff3:
  • Cmd: rm -f /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.pfam-A.gff3
  • Python: function get_hmmscan_gff3_task.cmd
• [x] TransDecoder.Predict:edulis.Trinity.fasta:
  • Cmd: /home/schwarl/.conda/envs/py3.dammit/bin/TransDecoder.Predict -t /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta --retain_pfam_hits /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.transdecoder_dir/longest_orfs.pep.x.pfam.tbl
• [x] cmscan:edulis.Trinity.fasta.x.Rfam.cm:
  • Cmd: cat /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta | /home/schwarl/.conda/envs/py3.dammit/bin/parallel --round-robin --pipe -L 2 -N 10000 --gnu -j 20 -a /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta /home/schwarl/.conda/envs/py3.dammit/bin/cmscan --cpu 1 --rfam --nohmmonly -E 1e-05 --tblout /dev/stdout -o /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.rfam.tbl.cmscan.out /home/schwarl/.dammit/databases/Rfam.cm /dev/stdin > /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.rfam.tbl
• [x] cmscan-gff3:edulis.Trinity.fasta.x.rfam.gff3:
  • Cmd: rm -f /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.rfam.gff3
  • Python: function get_cmscan_gff3_task.cmd
• [ ] lastal:/work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.OrthoDB.maf:
  • Python: function setup_profiler.add_profile_actions.start_profiling
  • Cmd: cat /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta | /home/schwarl/.conda/envs/py3.dammit/bin/parallel --round-robin --pipe -L 2 -N 10000 --gnu -j 20 -a /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta /home/schwarl/.conda/envs/py3.dammit/bin/lastal -F15 -D100000.0 /home/schwarl/.dammit/databases/aa_seq_euk.fasta > /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.OrthoDB.maf
  • Python: function setup_profiler.add_profile_actions.stop_profiling TaskError - taskid:lastal:/work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.OrthoDB.maf Command error: 'cat /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta | /home/schwarl/.conda/envs/py3.dammit/bin/parallel --round-robin --pipe -L 2 -N 10000 --gnu -j 20 -a /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta /home/schwarl/.conda/envs/py3.dammit/bin/lastal -F15 -D100000.0 /home/schwarl/.dammit/databases/aa_seq_euk.fasta > /work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.OrthoDB.maf' returned 255

######################################## TaskError - taskid:lastal:/work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.OrthoDB.maf lastal:/work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.OrthoDB.maf : parallel: Warning: A record was longer than 1048576. Increasing to --blocksize 1363150. parallel: Warning: A record was longer than 1363150. Increasing to --blocksize 1772096. parallel: Warning: A record was longer than 1772096. Increasing to --blocksize 2303726. parallel: Warning: A record was longer than 2303726. Increasing to --blocksize 2994845. parallel: Warning: A record was longer than 2994845. Increasing to --blocksize 3893300. parallel: Warning: A record was longer than 3893300. Increasing to --blocksize 5061291. parallel: Warning: A record was longer than 5061291. Increasing to --blocksize 6579680. parallel: Warning: A record was longer than 6579680. Increasing to --blocksize 8553585. parallel: Warning: A record was longer than 8553585. Increasing to --blocksize 11119662. parallel: Warning: A record was longer than 11119662. Increasing to --blocksize 14455562. parallel: Error: Output is incomplete. Cannot append to buffer file in /tmp. Is the disk full? parallel: Error: Change $TMPDIR with --tmpdir or use --compress. Warning: unable to close filehandle properly: No space left on device during global destruction.

lastal:/work/schwarl/alignments_trinity/edulis.Trinity.fasta.dammit/edulis.Trinity.fasta.x.OrthoDB.maf :

where do I use the --compress flag to fix this? in the annotate command? (I'm a newbie to annotation, so sorry if this is a stupid question!)