err code: -7

[lz870718]

Hi, I was trying to run spades in linux, but got following error:

== Error == system call for: "['/gpfs/home/user/.conda/envs/tormes-environment/share/spades-3.13.0-0/bin/spades-hammer', '/gpfs/data/projects/Denovo_result/corrected/configs/config.info']" finished abnormally, err code: -7

Could you please guide me through it? Thank you so much!

Zhi

[asl]

Sure. Just two lines below the ones you posted and one line above the github link there was a suggestion to provide certain files.

Will you please provide them?

[lz870718]

Hi Asl,

I sent them to [email protected]. Thank you for looking into it!

Zhi

[efogarty11]

Hello @asl! I am having a similar issue:

This is the command that I ran on our cluster:

/project2/meren/RESOURCES/PUBLIC/SOFTWARE/SPAdes-3.13.0-Linux/bin/metaspades.py	\
-o /project2/meren/PROJECTS/02_METASPADES \
--meta \
-1 /project2/meren/PROJECTS/01_QC/sewage_hic_0001_WW01_shotgun-QUALITY_PASSED_R1.fastq.gz \
-2 /project2/meren/PROJECTS/01_QC/sewage_hic_0001_WW01_shotgun-QUALITY_PASSED_R2.fastq.gz \
-t 30 \
-m 750

And this is the error:

== Error ==  system call for: 
"['/project2/meren/RESOURCES/PUBLIC/SOFTWARE/SPAdes-3.13.0-Linux/bin/spades-core',
'/project2/meren/PROJECTS/02_METASPADES/K55/configs/config.info', 
'/project2/meren/PROJECTS/02_METASPADES/K55/configs/mda_mode.info', 
'/project2/meren/PROJECTS/02_METASPADES/K55/configs/meta_mode.info']" finished abnormally, 
err code: -7

These are my spades.log and params.log files: params.txt spades.log

This is an assembly of a single metagenome from a wastewater treatment plant and I can't figure out why my assembly keeps failing. If you have any idea of whats going on from the attached files I'd love to hear them. Thank you so much!!

[asl]

Does the problem reproduce with SPAdes 3.14.0?

[asl]

@tania-k Please do not hijack the other issues as the problem might be completely different (as in your case). What is the filesystem on your /bigdata/ storage?

[efogarty11]

Hi @asl, I'm still waiting for the assembly to finish (it's been running for over 2 weeks) but is continuing to write to the spades.log file, so I'm hopeful that it's working! Thank you for the 3.14.0 suggestion!

[efogarty11]

Hi @asl, well funny timing, it just failed again using 3.14.0.

Here is the command that I ran:

metaspades.py -o 02_METASPADES/ --meta -1 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_1.fastq -2 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_2.fastq -t 30 -m 1500

This is the bottom of the spades.log file:

== Error ==  system call for: "['/automounts/bioware/bioware/linuxOpteron/spades-3.14.0/bin/spades-core', '/automounts/storage2/storage2/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/K55/configs/config.info', '/automounts/storage2/storage2/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/K55/configs/mda_mode.info', '/automounts/storage2/storage2/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/K55/configs/meta_mode.info']" finished abnormally, OS return value: 255

In case you have troubles running SPAdes, you can write to [email protected]
or report an issue on our GitHub repository github.com/ablab/spades
Please provide us with params.txt and spades.log files from the output directory.

SPAdes log can be found here: /automounts/storage2/storage2/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/spades.log

Thank you for using SPAdes!

I've attached the spades.log file and the params.txt. Thank you so much! params.txt spades.log

Also, if there is a way to fix this, is it possible to run it starting just from K55 and using the previously computed K21 and 33?

[asl]

So, the log reads:

116:35:28.875   179G / 721G  INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices
116:35:30.490   180G / 721G  ERROR   General                 (mmapped_reader.hpp        :  52)   mmap(2) failed. Reason: Cannot allocate memory. Error code: 12
116:35:30.490   180G / 721G  ERROR   General                 (mmapped_reader.hpp        :  52)   mmap(2) failed. Reason: Cannot allocate memory. Error code: 12

And you're running out of RAM. Use --continue / --restart-from options

[efogarty11]

Ahh, ok. I will try it out allocating more memory with the -m flag. Thank you!!

[efogarty11]

@asl I'm running into the issue that it isn't able to recognize my params.txt file. With both the --continue and the --restart-from flags, I get this error:

command line: perl /bioware/seqinfo/bin/clusterize -n 30 -log restart_assembly_sewage_hic_0001_WW01_shotgun.log metaspades.py -o 02_METASPADES/ --meta -1 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_1.fastq -2 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_2.fastq -t 35 -m 1900 --continue
Running on host: barhal-01.bpcservers.private
Running command: metaspades.py -o 02_METASPADES/ --meta -1 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_1.fastq -2 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_2.fastq -t 35 -m 1900 --continue


== Error ==  failed to parse command line of the previous run (/automounts/storage2/storage2/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/02_METASPADES/params.txt not found)!


In case you have troubles running SPAdes, you can write to [email protected]
or report an issue on our GitHub repository github.com/ablab/spades
Please provide us with params.txt and spades.log files from the output directory.
Job exit status: 1
Warning: job exited non-zero: 1

This file is in the directory that I'm running the assembly in, but for some reasons it thinks that the path to this file is /automounts/storage2/storage2/..../02_METASPADES/params.txt, when it is really /automounts/storage2/data/..../02_METASPADES/params.txt

Is there a way for me to explicitly tell metaspades where the params.txt file is?

[asl]

Well... I guess the problem is your wrapper script that hacks on cmdline of spades.py / metaspades.py and its working directory... Maybe specifying absolute path for -o will help.

[efogarty11]

Thank you for the suggestion @asl. I've tried as many variations as I can think of to try to restart it from there. I've tried --continue, --restart-from k55, and --restart-from last. When I put the full path for everything and run this:

/bioware/spades-3.14.0/bin/metaspades.py -o /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/ --meta -1 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_1.fastq -2 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_2.fastq -t 35 -m 1900 --restart-from mc

I get this error:

/automounts/storage2/storage2/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/ckH8v8myfW.sh
command line: perl /bioware/seqinfo/bin/clusterize -n 35 -log restart_assembly_sewage_hic_0001_WW01_shotgun.log /bioware/spades-3.14.0/bin/metaspades.py -o /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/ --meta -1 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_1.fastq -2 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_2.fastq -t 35 -m 1900 --restart-from last
Running on host: barhal-01.bpcservers.private
Running command: /bioware/spades-3.14.0/bin/metaspades.py -o /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/ --meta -1 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_1.fastq -2 /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/sewage_hic_0001_WW01_shotgun_2.fastq -t 35 -m 1900 --restart-from last
Traceback (most recent call last):
  File "/bioware/spades-3.14.0/bin/metaspades.py", line 639, in <module>
    main(sys.argv)
  File "/bioware/spades-3.14.0/bin/metaspades.py", line 577, in main
    cfg, dataset_data, command_line = parse_args(args, log)
  File "/bioware/spades-3.14.0/bin/metaspades.py", line 237, in parse_args
    secondary_filling=False, restart_from=False)
  File "/automounts/bioware/bioware/linuxOpteron/spades-3.14.0/share/spades/spades_pipeline/options_parser.py", line 1045, in parse_args
    options_storage.args, argv = parser.parse_known_args(options)
  File "/bioware/python-3.6.5-201804120909/lib/python3.6/argparse.py", line 1762, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/bioware/python-3.6.5-201804120909/lib/python3.6/argparse.py", line 1968, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/bioware/python-3.6.5-201804120909/lib/python3.6/argparse.py", line 1908, in consume_optional
    take_action(action, args, option_string)
  File "/bioware/python-3.6.5-201804120909/lib/python3.6/argparse.py", line 1836, in take_action
    action(self, namespace, argument_values, option_string)
  File "/automounts/bioware/bioware/linuxOpteron/spades-3.14.0/share/spades/spades_pipeline/options_parser.py", line 123, in __call__
    support.add_to_dataset(opt, arg, namespace.dataset_data)
  File "/automounts/bioware/bioware/linuxOpteron/spades-3.14.0/share/spades/spades_pipeline/support.py", line 538, in add_to_dataset
    if record_id not in dataset_data:  # setting default values for a new record
TypeError: argument of type 'NoneType' is not iterable
Job exit status: 1
Warning: job exited non-zero: 1

[olga24912]

Hi, @efogarty11! Thank you for detecting some problems with spades!

Please, can you follow next steps:

1. If it is possible, please, copy the output folder with results to a backup folder. It is helpful for making some experiments with running spades without fear to lose intermediate results. cp /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/ /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES_backup/

2. try to run /bioware/spades-3.14.0/bin/metaspades.py --restart-from last --checkpoints last -o /storage2/data/efogarty-analysis/GUT_PLASMIDS/HI-C/sewage/assembly_sewage_hic_0001_WW01_shotgun/02_METASPADES/

you need not write any extra parameters for restart-from except -o, otherwise spades try to update parameters.

[efogarty11]

thank you @olga24912! I made the back up but that seems like it's working! :) thanks again!!