Vireo error: _multiprocessing.cpython-39-x86_64-linux-gnu.so: failed to map segment from shared object

[bednarsky]

Dear hadge authors,

thanks again for your help in advance!

I am running the genetic demultiplexing pipeline for multiple samples, with a sample sheet modelled after the example you provided.

After a long time running, the pipeline fails like this:

[f2/65ada4] process > run_multi:gene_demultiplexi... [100%] 102 of 102 ✔ [21/78be35] process > run_multi:gene_demultiplexi... [100%] 194 of 194 [- ] process > run_multi:gene_demultiplexi... - [86/5049f3] process > run_multi:gene_demultiplexi... [100%] 98 of 98 [- ] process > run_multi:gene_demultiplexi... - [0d/38dc3b] process > run_multi:gene_demultiplexi... [ 4%] 4 of 91, failed: 1 [dd/a9251b] process > run_multi:gene_demultiplexi... [100%] 98 of 98 [ea/a993d7] process > run_multi:gene_demultiplexi... [100%] 99 of 99 [- ] process > run_multi:gene_demultiplexi... - ERROR ~ Error executing process > 'run_multi:gene_demultiplexing:demultiplex_vireo:vireo (4)'

Caused by: Process run_multi:gene_demultiplexing:demultiplex_vireo:vireo (4) terminated with an error exit status (1)

Command executed:

mkdir vireo_id418 mkdir vireo_id418/vireo_out touch vireo_id418/params.csv echo -e "Argument,Value cell_data,cellSNP.cells n_donor,2 donor_data,Donor file is not given genoTag,GT noDoublet,False nInit,50 extraDonor,0 extraDonorMode,distance learnGT,False ASEmode,False noPlot,False randSeed,None cellRange,all callAmbientRNAs,False nproc,2" >> vireo_id418/params.csv

     vireo -c cellSNP.cells.vcf -N 2    --nInit 50 --extraDonor 0                    --nproc 2 -o vireo_id418/vireo_out

     if ([ "Donor file is not given" = "Donor file is not given" ]); then
         if ([ "False" = "True" ] || [ "False" = "vireo" ]); then
             GTbarcode -i vireo_id418/vireo_out/GT_donors.vireo.vcf.gz -o vireo_id418/vireo_out/filtered_variants.tsv None
         fi
     fi

Command exit status: 1

Command output: [vireo] Loading cell VCF file ... [vireo] Demultiplex 10372 cells to 2 donors with 175894 variants.

Command error: Traceback (most recent call last): File "/path/to/fileresources/demultiplexing/hadge/work/conda/env-32552abe9d1b7c018779365095e3cabe/bin/vireo", line 11, in sys.exit(main()) File "/path/to/file/resources/demultiplexing/hadge/work/conda/env-32552abe9d1b7c018779365095e3cabe/lib/python3.9/site-packages/vireoSNP/vireo.py", line 201, in main res_vireo = vireo_wrap(cell_dat['AD'], cell_dat['DP'], n_donor=n_donor, File "/path/to/file/resources/demultiplexing/hadge/work/conda/env-32552abe9d1b7c018779365095e3cabe/lib/python3.9/site-packages/vireoSNP/utils/vireo_wrap.py", line 76, in vireo_wrap pool = multiprocessing.Pool(processes = nproc) File "/path/to/file/resources/demultiplexing/hadge/work/conda/env-32552abe9d1b7c018779365095e3cabe/lib/python3.9/multiprocessing/context.py", line 118, in Pool from .pool import Pool File "/path/to/file/resources/demultiplexing/hadge/work/conda/env-32552abe9d1b7c018779365095e3cabe/lib/python3.9/multiprocessing/pool.py", line 30, in from .connection import wait File "/path/to/file/resources/demultiplexing/hadge/work/conda/env-32552abe9d1b7c018779365095e3cabe/lib/python3.9/multiprocessing/connection.py", line 21, in import _multiprocessing ImportError: /path/to/file/resources/demultiplexing/hadge/work/conda/env-32552abe9d1b7c018779365095e3cabe/lib/python3.9/lib-dynload/_multiprocessing.cpython-39-x86_64-linux-gnu.so: failed to map segment from shared object

This happens even though the pipeline finished the vireo process for a few samples, so that I can see the expected output files (as described in documentation, except some of the plots). Any idea what's going wrong?

GPT4 suggested this, but I am not sure what actually to do:

GPT4's opinion on this error

The error message indicates that the `vireo` process within the Nextflow pipeline has terminated with an error exit status (1). The specific error is an `ImportError` that occurred when trying to import the `_multiprocessing` module in Python. The error message states that the shared object file `_multiprocessing.cpython-39-x86_64-linux-gnu.so` could not be mapped.

This shared object file is a dynamic library that should be part of the Python installation in the conda environment being used (env-32552abe9d1b7c018779365095e3cabe). The error message suggests that there is a problem with this shared object file, which could be due to several reasons:

1. File Permissions: The file may not have the correct permissions set, preventing it from being read or executed.

2. Corrupted File: The shared object file might be corrupted or incomplete.

3. Filesystem Issues: The error "failed to map segment from shared object" can sometimes occur if there is not enough memory available to map the file, or if there are restrictions on the filesystem where the file is located (e.g., a noexec mount option).

4. Conda Environment: There might be an issue with the conda environment itself, such as a broken environment or an incorrect setup.

To resolve this issue, you may need to:

• Check the permissions of the shared object file and ensure it is readable and executable.
• Verify that there is enough memory available on the system to load the shared object file.
• Check if the filesystem where the conda environment is located has any restrictions (e.g., mounted with noexec).
• Reinstall the conda environment to ensure all dependencies and shared libraries are correctly installed and not corrupted.

I tried to resume the pipeline with the -resume flag in the nextflow command and then a few more samples finish the vireo process, but the pipeline dies again at the same sample. I tried to resort the sample_sheet, but then the pipeline just dies at another sample.

Thanks!