Segmentation fault in spades-hammer (error -11)

[caspargross]

Spades does not run on my local Laptop. The spades-hammer step exits with error code -11 (Segfault) Similar issue: https://github.com/ablab/spades/issues/191

To narrow it down I ran the following Spades config and get the same error. spades.py --test -t 4 -m 4

Here is the coredump:

           PID: 1224 (spades-hammer)
           UID: 1000 (caspar)
           GID: 100 (users)
        Signal: 11 (SEGV)
     Timestamp: Wed 2018-10-17 09:22:19 CEST (54s ago)
  Command Line: /home/caspar/.conda/envs/spades_env/share/spades-3.12.0-1/bin/spades-hammer /home/caspar/Documents/Uni/MasterThesis/pipelines/hybridAssembly/testOut/Reports/spades_test/corrected/configs/config.info
    Executable: /home/caspar/.conda/envs/spades_env/share/spades-3.12.0-1/bin/spades-hammer
 Control Group: /user.slice/user-1000.slice/[email protected]/gnome-terminal-server.service
          Unit: [email protected]
     User Unit: gnome-terminal-server.service
         Slice: user-1000.slice
     Owner UID: 1000 (caspar)
       Boot ID: da8a9442e9db435796d56b2bfb3254b7
    Machine ID: f60699ddd6ee4f019374f457d3167c3f
      Hostname: combin
       Storage: /var/lib/systemd/coredump/core.spades-hammer.1000.da8a9442e9db435796d56b2bfb3254b7.1224.1539760939000000.lz4
       Message: Process 1224 (spades-hammer) of user 1000 dumped core.
                
                Stack trace of thread 1224:
                #0  0x00000000005a5f59 n/a (/home/caspar/.conda/envs/spades_env/share/spades-3.12.0-1/bin/spades-hammer)
                #1  0x000000000060c7c0 n/a (/home/caspar/.conda/envs/spades_env/share/spades-3.12.0-1/bin/spades-hammer)
                #2  0x000000000040a709 n/a (/home/caspar/.conda/envs/spades_env/share/spades-3.12.0-1/bin/spades-hammer)
                #3  0x0000000000000001 n/a (n/a)
                #4  0x0000000100000002 n/a (n/a)

Attached files: params.txt spades.log

[asl]

Something looks incorrect at least with your Linux version: " OS: Linux-4.18.9-arch1-1-ARCH-x86_64-with-arch". Why is it so?

Does the problem reproduce with our pre-built binaries?

[openpaul]

I have the same error code while running metaspades.

I tried the same with the original precompiled version 3.13 and with the conda version (3.12) and the error persists.

---- edit ---- it seems to be working on version 3.11 (bioconda install).

[crysclitheroe]

Hi there

Is this error a memory problem? On Linux cluster, (max 600GB mem, 4 nodes, 8 cpuspernode) I also get error code -11 on SPAdes 3.13.0:

== Error == system call for: "['/work/.apps/free72/SPAdes/3.13.0/bin/spades-core', '/path/to/assembly/K77/configs/config.info']" finished abnormally, err code: -11

on the "hybrid_aligning.cpp" step for k77 (eukaryotic de novo assembly using long (2.7GB nanopore) and short reads (120GB, paired-end Illumina, carefully cleaned)); 4:42:39.180 59G / 85G INFO General (hybrid_aligning.cpp : 268) Prepared batch 0 of 50000 reads.

I will try spades 3.11.1 in the mean time.

[asl]

@Tipplynne Please do not hijack the other issues. Please open a new one attaching the relevant data (spades.log and params.txt file).

[crysclitheroe]

Dear @asl

Thanks for getting back so fast!

My apologies, I didn't mean to intrude on the issue, I though it would go here because of the same error code. I will of course start a new issue.

Thanks for all the hard work!

[caspargross]

Something looks incorrect at least with your Linux version: " OS: Linux-4.18.9-arch1-1-ARCH-x86_64-with-arch". Why is it so?

Why does it look incorrect? Its just standard arch kernel, no custom changes.

Does the problem reproduce with our pre-built binaries?

Yes. But it is only on my machine, so I suppose there is something wrong on my part. Workaround is to just use another machine. If you want we can close this issue.

[asl]

@caspargross Is there anything in the system logs? All this looks a bit strange...

[indexofire]

install by conda with spades-3.13, same error as above. downgrade to 3.11, everything is fine now.

[asl]

@indexofire We neither maintain, nor support conda packages. Please contact conda maintainers, it's likely something is broken there.

[asl]

@notestaff Do you know what might be the reason?

[dswan]

+1 for this issue, using pre-compiled binaries and metaspades I have a dataset that processes fine with 3.11 but spades-hammer segfaults with 3.12 and 3.13. Happy to provide more info once my 3.11 run finishes.

[andreott]

Same problem here, spades.py --test producing the error for 3.13 (same for conda installed and downloaded pre-built)

Attached files: spades.log params.txt

[asl]

We do not support nor maintain conda packages. Please report this problem to conda or compile SPAdes from source.

[andreott]

I edited/clarified my post. The problem occurs for the conda installed version and also for the pre-built from the Spades site.

[asl]

It certainly may happen as these days it is impossible to provide pre-built binaries that could be run everywhere (sadly). Please consider building from source.

[notestaff]

I'm trying to change the conda recipe to build spades from source.

Are the compilation warnings at https://circleci.com/gh/bioconda/bioconda-recipes/69783 safe to ignore?

On Fri, Aug 16, 2019 at 6:07 AM Anton Korobeynikov [email protected] wrote:

It certainly may happen as these days it is impossible to provide pre-built binaries that could be run everywhere (sadly). Please consider building from source.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[notestaff]

I get a config dir error from spades.py --test when building from source: https://circleci.com/gh/bioconda/bioconda-recipes/69783

On Fri, Aug 16, 2019 at 6:07 AM Anton Korobeynikov [email protected] wrote:

It certainly may happen as these days it is impossible to provide pre-built binaries that could be run everywhere (sadly). Please consider building from source.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ablab/spades/issues/194?email_source=notifications&email_token=AALSMY2ZSTR6PIMSGQONP5DQEZ35RA5CNFSM4F4NGA62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4OHMNA#issuecomment-521958964, or mute the thread https://github.com/notifications/unsubscribe-auth/AALSMY26MY4BAK5FWAWFFTLQEZ35RANCNFSM4F4NGA6Q .

[asl]

@notestaff For some reason it is using external boost, not the pre-packaged one. You really need either to patch the external boost (the bug there is sitting uncommitted for ages) or use the pre-packaged one. Unsetting Boost_ROOT for cmake might help.

[notestaff]

Thanks!

On MacOS I get errors saying the include parallel/algorithms is missing: https://circleci.com/gh/bioconda/bioconda-recipes/69820 Any suggestions on how to fix this?

On Fri, Aug 16, 2019 at 3:20 PM Anton Korobeynikov [email protected] wrote:

@notestaff https://github.com/notestaff For some reason it is using external boost, not the pre-packaged one. You really need either to patch the external boost (the bug there is sitting uncommitted for ages) or use the pre-packaged one. Unsetting Boost_ROOT for cmake might help.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ablab/spades/issues/194?email_source=notifications&email_token=AALSMY5ZAMUURABG5NEFSJ3QE34YVA5CNFSM4F4NGA62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4PPLAQ#issuecomment-522122626, or mute the thread https://github.com/notifications/unsubscribe-auth/AALSMYZVZD7S5CLI75LQGTLQE34YVANCNFSM4F4NGA6Q .

[asl]

@notestaff Are you building with non-Apple clang with OpenMP enabled?

[notestaff]

@asl yes, one of the dependencies of this recipe is llvm-openmp, from https://github.com/conda-forge/openmp-feedstock/tree/master/recipe

[asl]

Oh, then you'd need to patch the build scripts a bit:

commit cb33372a40645fd179ceb95d273c085134c26239
Author: Anton Korobeynikov <[email protected]>
Date:   Thu Jul 18 00:20:08 2019 +0300

    Ensure we're using parallel stdlibc++ when we're compiling for it

diff --git a/assembler/src/cmake/flags.cmake b/assembler/src/cmake/flags.cmake
index a3f81567cc..1da9146dc2 100644
--- a/assembler/src/cmake/flags.cmake
+++ b/assembler/src/cmake/flags.cmake
@@ -6,7 +6,9 @@ if (OPENMP_FOUND)
   set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${OpenMP_CXX_FLAGS}")
   set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${OpenMP_EXE_LINKER_FLAGS}")
 # Use parallel libstdc++ if possible
-  add_definitions(-DUSE_GLIBCXX_PARALLEL=1)
+  if (NOT "${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang")
+    add_definitions(-DUSE_GLIBCXX_PARALLEL=1)
+  endif()
 else ()
   if (NOT "${CMAKE_CXX_COMPILER_ID}" MATCHES "Clang")
     message(FATAL_ERROR "SPAdes requires OpenMP to be available")

[notestaff]

Thanks @asl . Is the binary MacOS distribution from http://cab.spbu.ru/files/release3.13.0/SPAdes-3.13.0-Darwin.tar.gz built with -DUSE_GLIBCXX_PARALLEL=1 ? Is that expected to make a significant difference?

Also, at http://cab.spbu.ru/software/spades/ the download link to MacOS binaries for 3.13.1 is broken.

[notestaff]

@andreott @caspargross @dswan @indexofire @openpaul can you try with the updated conda package?

[asl]

Is that expected to make a significant difference?

It was built by gcc with OpenMP enabled. There will be some difference, but not that much (the version from website being faster sometimes).

Also, at http://cab.spbu.ru/software/spades/ the download link to MacOS binaries for 3.13.1 is broken.

Oh, indeed. Though, all the files are there and direct link works. Thanks!

[matrs]

Can you post this clarification in the README please?, all the people using Arch can avoid loosing time searching for this information. I just spent more than an hour on this to finally arrive to these posts, a small line clarifying that in the main README would be really useful.

[caspargross]

Old and stale. Closing