ecotyper
ecotyper copied to clipboard
CIBERSORTx integration fails
Hello there once more :-)
As I am aiming to become a regular customer, I have yet again a new issue 😄. Technically it's a CIBERSORTx issue not an EcoTyper issue, but perhaps the forum here can help solve it quicker.
I have finally received my token for CIBERSORTx 🥳, but alas the pipeline fails due to Segmentation Faults from the singularity image 🥺.
Here is some background:
I wish to perform EcoType discovery from bulk-RNASeq data using EcoTyper and CIBERSORTx.
On our cluster we only have Singularity. Hence, I converted the docker images to SIF using
singularity build cibersortx_fractions.sif docker://cibersortx/fractions
Next, I specified the appropriate path to the cibersortx_fractions.sif
in the EcoTyper config file and submitted the job.
I am using just the default Carcinoma
setting for CIBERSORTx. However, it appears that some temporary files cannot be written, which causes the JobQueue() to fail repeatedly until the entire CIBERSORTx step crashes.
The repeated "error" is:
Warning message:
In fread(X_file, header = F, sep = "\t") :
File '/src/outdir//temp.Fractions.coreSVR.X.tsv' has size 0. Returning a NULL data.table.
Warning message:
In fread(Y_file, header = F, sep = "\t") :
File '/src/outdir//temp.Fractions.coreSVR.Y.tsv' has size 0. Returning a NULL data.table.
Error: $ operator is invalid for atomic vectors
In addition: Warning message:
In mclapply(1:length(nus), res, Y[, j], mc.cores = length(nus)) :
all scheduled cores encountered errors in user code
Execution halted
which ultimately leads a segmentation fault error as:
sh: line 10: 62182 Segmentation fault singularity exec -c -B /data/users/noahkleinschmidt/EcoTyper/scratchdir/:/tmp -B /data/users/noahkleinschmidt/EcoTyper/ecotyper/CIBERSORTx/fractions/discovery/TCGA_bulk_discovery/LM22/B_mode/:/src/data -B /data/users/noahkleinschmidt/EcoTyper/ecotyper/CIBERSORTx/fractions/discovery/TCGA_bulk_discovery/LM22/B_mode/:/src/outdir /data/users/noahkleinschmidt/EcoTyper/cibersortx/cibersortx_fractions.sif /src/CIBERSORTxFractions --username [email protected] --token 75d719fbb04eb7eae1705c75515c4db9 --mixture /src/data/mixture.txt --sigmatrix /src/data/sigmatrix.txt --sourceGEPs /src/data/sourceGEPs.txt --rmbatchBmode TRUE --verbose TRUE
Error:
Execution halted
Error:
Execution halted
Error in RunJobQueue() :
EcoTyper failed. Please check the error message above!
Execution halted
The full logs are available via pastecode.io.
You may notice from the command behind the segmentation fault, this is already not 100% the original command. Here are some things I have tried to fix this:
- (re-)setting
SINGULARITY_TMPDIR
/SINGULARITY_CACHEDIR
/SINGULARITY_LOCALCACHEDIR
(in any combination, both in the terminal before submitting the slurm job, and as part of the EcoTyper internal call to CIBERSORTx), all to no effect. - manually binding some directory as
/tmp/
(our cluster has a tmp quota, the log from above is from this one), but no effect either. - raising the
ulimit
settings tounlimited
- no effect. - pulling the docker image on my local laptop (MacOS) and building the singularity image there using sudo - no effect.
- while trying this, I also tried re-assigning my
local SINGULARITY_{}
variables during the build as I understand that _TMPDIR and _CACHEDIR are only used for building. - the uploaded SIF images did not behave any different from the ones built on the cluster directly.
- in fact, they did raise the same segmentation fault when running on my local laptop
- while trying this, I also tried re-assigning my
- building an entirely new image by first making a
sandbox
from the docker image (just creates a folder hierarchy), then converting that to SIF - no effect. - in this context I also tried writing a wrapper script in the sandbox to first increaseulimit
settings before calling on theCIBERSORTxFractions
executable. Convert sandbox to SIF, let EcoTyper call the wrapper instead. I just got the error thatulimit
modifications were not allowed, so no effect too. - building a docker
sandbox
on my laptop and running theCIBERSORTxFractions
directly- this showed that the
temp.{X|Y}.tsv
files actually did appear inoutdir
but only for a brief moment until they disappeared again, followed by the same segmentation fault.
- this showed that the
Something I have not yet tried is altering the EcoTyper CIBERSORTx cmd_line
to directly call the CIBERSORTxFractions
executable within /src/
in my sandbox instead of via singularity. I will try this and post an update.
I have seen that virtually the same issue was already described before but without solution. Thanks a lot already for any inputs 🌻
May I ask a question irrelevant to this issue?How long did it take to receive the token? I've been waiting for a month...
It was around the same time for me 😅. I emailed the developers a couple of times, but things just take time, I assume. I'm sure they have not forgotten about you, though 🌻