quantms icon indicating copy to clipboard operation
quantms copied to clipboard
verified publisher icon bigbio

Error memory TMT dataset PXD012574

Open ypriverol opened this issue 3 years ago • 5 comments

Description of the bug

URL of the working file: http://ftp.pride.ebi.ac.uk/pub/databases/pride/resources/proteomes/proteogenomics/benchmakrs/15acadc82868c799e567dd1b219b44/


nf-core/quantms execution completed unsuccessfully!

The exit status of the task that caused the workflow execution to fail was: 130.

The full error message was:

Error executing process > 'NFCORE_QUANTMS:QUANTMS:TMT:PROTEININFERENCE:PROTEININFERENCER'

Caused by:
  Process `NFCORE_QUANTMS:QUANTMS:TMT:PROTEININFERENCE:PROTEININFERENCER` terminated with an error exit status (130)

Command executed:

  ProteinInference \
      -in ID_mapper_merge.consensusXML \
      -threads 48 \
      -picked_fdr true \
      -picked_decoy_string DECOY_ \
      -protein_fdr true \
      -Algorithm:use_shared_peptides true \
      -Algorithm:annotate_indistinguishable_groups true \
       \
      -Algorithm:score_aggregation_method best \
      -Algorithm:min_peptides_per_protein 1 \
      -out ID_mapper_merge_epi.consensusXML \
      -debug 0 \
      |& tee ID_mapper_merge_inference.log
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_QUANTMS:QUANTMS:TMT:PROTEININFERENCE:PROTEININFERENCER":
      ProteinInference: $(ProteinInference 2>&1 | grep -E '^Version(.*) ' | sed 's/Version: //g' | cut -d ' ' -f 1)
  END_VERSIONS

Command exit status:
  130

Command output:
  Loading input...
  Loading input took 41.49 s (wall), 41.40 s (CPU), 0.63 s (system), 40.77 s (user)
   occurred 2 times
  Merging IDs across runs...
  Merging IDs across runs took 0.00 s (wall), 0.00 s (CPU), 0.00 s (system), 0.00 s (user)
  Aggregating protein scores...
  Building graph on 738370 features, 18751 unassigned spectra (if chosen) and 22415 proteins.
  Progress of 'Building graph...':
  
  -- done [took 0.66 s (CPU), 0.66 s (Wall)] -- 
  Progress of 'Annotating indistinguishable proteins...':
  
  -- done [took 1.23 s (CPU), 0.07 s (Wall)] -- 
  Found 17785 connected components.
  Aggregating protein scores took 0.00 s (wall), 0.00 s (CPU), 0.00 s (system), 0.00 s (user)
  Calculating target-decoy q-values...
  Storing output...
  ConsensusXMLFile::store():  found 1 invalid unique ids

Command wrapper:
  Aggregating protein scores...
  Building graph on 738370 features, 18751 unassigned spectra (if chosen) and 22415 proteins.
  Progress of 'Building graph...':
  
  -- done [took 0.66 s (CPU), 0.66 s (Wall)] -- 
  Progress of 'Annotating indistinguishable proteins...':
  
  -- done [took 1.23 s (CPU), 0.07 s (Wall)] -- 
  Found 17785 connected components.
  Aggregating protein scores took 0.00 s (wall), 0.00 s (CPU), 0.00 s (system), 0.00 s (user)
  Calculating target-decoy q-values...
  Storing output...
  ConsensusXMLFile::store():  found 1 invalid unique ids
  
  ------------------------------------------------------------
  Sender: LSF System 
  Subject: Job 7751604:  in cluster  Exited
  
  Job  was submitted from host  by user  in cluster  at Thu Nov 24 18:00:09 2022
  Job was executed on host(s) <48*hl-codon-bm-07>, in queue , as user  in cluster  at Thu Nov 24 18:00:19 2022
   was used as the home directory.
   was used as the working directory.
  Started at Thu Nov 24 18:00:19 2022
  Terminated at Thu Nov 24 18:10:28 2022
  Results reported at Thu Nov 24 18:10:28 2022
  
  Your job looked like:
  
  ------------------------------------------------------------
  # LSBATCH: User input
  ------------------------------------------------------------
  
  TERM_MEMLIMIT: job killed after reaching LSF memory usage limit.
  Exited with exit code 130.
  
  Resource usage summary:
  
      CPU time :                                   612.00 sec.
      Max Memory :                                 318248 MB
      Average Memory :                             151518.11 MB
      Total Requested Memory :                     307200.00 MB
      Delta Memory :                               -11048.00 MB
      Max Swap :                                   -
      Max Processes :                              13
      Max Threads :                                68
      Run time :                                   608 sec.
      Turnaround time :                            619 sec.
  
  The output (if any) is above this job summary.

Work dir:
  /hps/nobackup/juan/pride/reanalysis/differential-expression/tmt/PXD012574/work/39/15acadc82868c799e567dd1b219b44

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

Command used and terminal output

No response

Relevant files

No response

System information

No response

ypriverol avatar Nov 24 '22 19:11 ypriverol

Even with more memory, it continues failing:

Error executing process > 'NFCORE_QUANTMS:QUANTMS:TMT:PROTEININFERENCE:PROTEININFERENCER'

Caused by:
  Process `NFCORE_QUANTMS:QUANTMS:TMT:PROTEININFERENCE:PROTEININFERENCER` terminated with an error exit status (139)

Command executed:

  ProteinInference \
      -in ID_mapper_merge.consensusXML \
      -threads 48 \
      -picked_fdr true \
      -picked_decoy_string DECOY_ \
      -protein_fdr true \
      -Algorithm:use_shared_peptides true \
      -Algorithm:annotate_indistinguishable_groups true \
       \
      -Algorithm:score_aggregation_method best \
      -Algorithm:min_peptides_per_protein 1 \
      -out ID_mapper_merge_epi.consensusXML \
      -debug 0 \
      |& tee ID_mapper_merge_inference.log
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_QUANTMS:QUANTMS:TMT:PROTEININFERENCE:PROTEININFERENCER":
      ProteinInference: $(ProteinInference 2>&1 | grep -E '^Version(.*) ' | sed 's/Version: //g' | cut -d ' ' -f 1)
  END_VERSIONS

Command exit status:
  139

Command output:
  Loading input...
  Loading input took 41.83 s (wall), 41.62 s (CPU), 0.76 s (system), 40.86 s (user)
  <Loading input...> occurred 2 times
  Merging IDs across runs...
  Merging IDs across runs took 0.00 s (wall), 0.00 s (CPU), 0.00 s (system), 0.00 s (user)
  Aggregating protein scores...
  Building graph on 738370 features, 18751 unassigned spectra (if chosen) and 22415 proteins.
  Progress of 'Building graph...':
  
    9.92 %               
  -- done [took 0.59 s (CPU), 0.60 s (Wall)] -- 
  Progress of 'Annotating indistinguishable proteins...':
  
  -- done [took 2.60 s (CPU), 0.11 s (Wall)] -- 
  Found 17785 connected components.
  Aggregating protein scores took 0.00 s (wall), 0.00 s (CPU), 0.00 s (system), 0.00 s (user)
  Calculating target-decoy q-values...
  Storing output...
  ConsensusXMLFile::store():  found 1 invalid unique ids

Command error:
  .command.sh: line 15: 2292302 Segmentation fault      (core dumped) ProteinInference -in ID_mapper_merge.consensusXML -threads 48 -picked_fdr true -picked_decoy_string DECOY_ -protein_fdr true -Algorithm:use_shared_peptides true -Algorithm:annotate_indistinguishable_groups true -Algorithm:score_aggregation_method best -Algorithm:min_peptides_per_protein 1 -out ID_mapper_merge_epi.consensusXML -debug 0 2>&1
       2292303 Done                    | tee ID_mapper_merge_inference.log

Work dir:
  /hps/nobackup/juan/pride/reanalysis/differential-expression/tmt/PXD012574/work/b7/2d34857a791de24ebf9f93e632526e

Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh`

ypriverol avatar Nov 24 '22 20:11 ypriverol

I have run the dataset only with comet and it finished. @jpfeuffer

ypriverol avatar Nov 25 '22 08:11 ypriverol

Stacktrace:

Toggle me! 
#74641 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=-nan(0x8000000000000), num=@0x7ffc9e6e4220: inf, exp=315) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74642 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=308) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74643 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=301) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74644 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=294) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74645 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=287) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74646 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=280) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74647 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=273) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74648 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=266) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74649 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=259) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74650 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=252) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74651 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=245) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74652 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=238) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
...
#74684 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=14) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74685 0x00007f82c7ea57b5 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, num=@0x7ffc9e6e4220: inf, exp=7) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:597
#74686 0x00007f82c7ea3d90 in boost::spirit::karma::int_inserter<10u, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type> > (sink=..., n=inf) at /usr/include/boost/spirit/home/karma/numeric/detail/numeric_utils.hpp:656
#74687 0x00007f82c7ea26a1 in boost::spirit::karma::real_policies<double>::integer_part<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type> > (sink=..., n=inf, sign=false, force_sign=false) at /usr/include/boost/spirit/home/karma/numeric/real_policies.hpp:192
#74688 0x00007f82c7ea0643 in boost::spirit::karma::real_inserter<double, OpenMS::StringConversions::BK_PrecPolicy<double>, boost::spirit::unused_type, boost::spirit::unused_type>::call_n<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=inf, p=...) at /usr/include/boost/spirit/home/karma/numeric/detail/real_utils.hpp:164
#74689 0x00007f82c7e9e58b in boost::spirit::karma::real_policies<double>::call<boost::spirit::karma::real_inserter<double, OpenMS::StringConversions::BK_PrecPolicy<double>, boost::spirit::unused_type, boost::spirit::unused_type>, boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, OpenMS::StringConversions::BK_PrecPolicy<double> > (sink=..., n=4.6467477951396374e-310, p=...) at /usr/include/boost/spirit/home/karma/numeric/real_policies.hpp:94
#74690 0x00007f82c7e9d880 in boost::spirit::karma::real_inserter<double, OpenMS::StringConversions::BK_PrecPolicy<double>, boost::spirit::unused_type, boost::spirit::unused_type>::call<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, double> (sink=..., n=4.6467477951396374e-310, p=...) at /usr/include/boost/spirit/home/karma/numeric/detail/real_utils.hpp:56
#74691 0x00007f82c7e9c8af in boost::spirit::karma::any_real_generator<double, OpenMS::StringConversions::BK_PrecPolicy<double>, boost::spirit::unused_type, boost::spirit::unused_type>::generate<boost::spirit::karma::detail::output_iterator<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::unused_type>, boost::spirit::context<boost::fusion::cons<double const&, boost::fusion::nil_>, boost::spirit::locals<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na> >, boost::spirit::unused_type, double> (this=0x7ffc9e6e440f, sink=..., context=..., d=..., attr=@0x7ffc9e6e4488: 4.6467477951396374e-310) at /usr/include/boost/spirit/home/karma/numeric/real.hpp:192
#74692 0x00007f82c7e9bdf8 in boost::spirit::karma::generate<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, mpl_::int_<0>, boost::spirit::karma::real_generator<double, OpenMS::StringConversions::BK_PrecPolicy<double> >, double> (sink=..., expr=..., attr=@0x7ffc9e6e4488: 4.6467477951396374e-310) at /usr/include/boost/spirit/home/karma/generate.hpp:69
#74693 0x00007f82c7e9b1c0 in boost::spirit::karma::generate<std::back_insert_iterator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::spirit::karma::real_generator<double, OpenMS::StringConversions::BK_PrecPolicy<double> >, double> (sink_=..., expr=..., attr=@0x7ffc9e6e4488: 4.6467477951396374e-310) at /usr/include/boost/spirit/home/karma/generate.hpp:91
#74694 0x00007f82c7e9a540 in OpenMS::StringConversions::append (d=4.6467477951396374e-310, target=...) at /workspace/OpenMS/src/openms/include/OpenMS/DATASTRUCTURES/StringConversions.h:205
#74695 0x00007f82c7e98436 in OpenMS::String::String (this=0x7ffc9e6e45e0, d=4.6467477951396374e-310, full_precision=true) at /workspace/OpenMS/src/openms/source/DATASTRUCTURES/String.cpp:147

and then an infinite loop.

jpfeuffer avatar Nov 25 '22 11:11 jpfeuffer

@timosachsenberg

jpfeuffer avatar Nov 25 '22 11:11 jpfeuffer

see https://github.com/OpenMS/OpenMS/issues/6507

jpfeuffer avatar Nov 25 '22 12:11 jpfeuffer

Solved.

ypriverol avatar Mar 10 '25 07:03 ypriverol