bioinfo.sif is corrupted: wrong partition size

[Shrishtee-kandoi]

When I try to run the map_leafcutter_cluster_to_gene part for post-processing, it gives me this error..

FATAL:   could not open image /restricted/projectnb/casa/xqtl-pipeline/container/singularity/bioinfo.sif: SIF image /restricted/projectnb/casa/xqtl-pipeline/container/singularity/bioinfo.sif is corrupted: wrong partition size
/opt/conda/bin/python: can't open file '/restricted/projectnb/casa/skandoi/ROSMAP_DLPFC/scripts/tmp5dqppsvu/singularity_run_94692.py': [Errno 2] No such file or directory

Script used:

sos run /restricted/projectnb/casa/xqtl-pipeline/code/data_preprocessing/phenotype/gene_annotation.ipynb map_leafcutter_cluster_to_gene \
    --cwd /restricted/projectnb/casa/skandoi/ROSMAP_DLPFC/leafcutter_output/batch_all \
    --intron_count  /restricted/projectnb/casa/skandoi/ROSMAP_DLPFC/leafcutter_output/batch_all/leafcutter_bam_intron_usage_perind.counts.gz \
    --phenoFile /restricted/projectnb/casa/skandoi/ROSMAP_DLPFC/leafcutter_output/batch_all/leafcutter_bam_intron_usage_perind.counts.gz_raw_data.qqnorm.txt \
    --annotation-gtf /restricted/projectnb/casa/skandoi/ROSMAP_DLPFC/reference_data/Homo_sapiens.GRCh38.103.chr.reformatted.ERCC.gtf \
    --container /restricted/projectnb/casa/xqtl-pipeline/container/singularity/bioinfo.sif \
    --walltime 100h \
    --numThreads 20 \
    --mem 40G -J 50 -c csg.yml -q neurology

Log file:

INFO: Running [32mmap_leafcutter_cluster_to_gene[0m: 
[91mERROR[0m: [91mmap_leafcutter_cluster_to_gene (id=a02cf84eef1e061b) returns an error.[0m
[91mERROR[0m: [91m[map_leafcutter_cluster_to_gene]: [0]: Executing script in Singularity returns an error (exitcode=2, stderr=/restricted/projectnb/casa/skandoi/ROSMAP_DLPFC/reference_data/Homo_sapiens.GRCh38.103.chr.reformatted.ERCC.gtf.stderr).
The script has been saved to /usr4/bf527/skandoi/.sos/f62e6760de9b87f7//usr4/bf527/skandoi/.sos/f62e6760de9b87f7.To reproduce the error please run:
[0m[32msingularity exec  /restricted/projectnb/casa/xqtl-pipeline/container/singularity/bioinfo.sif python /usr4/bf527/skandoi/.sos/f62e6760de9b87f7/singularity_run_94692.py[0m[91m[0m

[Shrishtee-kandoi]

Update: It is successfully executed upon running it interactively but gives the same error when using bash scripts.

[hsun3163]

It is unclear what makes the difference despite the success. It could be that the SINGULARITY_BIND variable in the csg was not properly recognized, but it contradicts the previous success.

[grennfp]

Have there been any updates to this? I have run into a similar issue that does not resolve when I try running locally versus through a bash script.

Error:

/opt/conda/bin/python: can't open file '/restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/scripts/tmpdbgt_wz5/singularity_run_83472.py': [Errno 2] No such file or directory

Script:

#!/bin/bash -l

module load miniconda
conda activate xqtl_env2

sos run /restricted/projectnb/casa/frank/xqtl_project/xqtl-pipeline/pipeline/gene_annotation.ipynb map_leafcutter_cluster_to_gene \
    --cwd /restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/leafcutter_output/batch_all \
    --intron_count  /restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/leafcutter_output/batch_all/batch_all_bam_no_ext_intron_usage_perind.counts.gz  \
    --phenoFile /restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/leafcutter_output/batch_all/batch_all_bam_no_ext_intron_usage_perind.counts.gz_raw_data.qqnorm.txt \
    --annotation-gtf /restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/reference_data/Homo_sapiens.GRCh38.103.chr.reformatted.collapse_only.gene.ERCC.gtf \
    --container /restricted/projectnb/casa/frank/xqtl_project/xqtl-pipeline/container/singularity/bioinfo.sif \
    --walltime 100h \
    --numThreads 20 \
    --mem 40G -J 50 -c csg.yml -q neurology

Log:

INFO: Running [32mmap_leafcutter_cluster_to_gene[0m: 
[91mERROR[0m: [91mmap_leafcutter_cluster_to_gene (id=5a76d510fef6330f) returns an error.[0m
[91mERROR[0m: [91m[map_leafcutter_cluster_to_gene]: [0]: Executing script in Singularity returns an error (exitcode=2, stderr=/restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/reference_data/Homo_sapiens.GRCh38.103.chr.reformatted.collapse_only.gene.ERCC.gtf.stderr).
The script has been saved to /usr3/graduate/fgrennjr/.sos/4966833ed02ca234//usr3/graduate/fgrennjr/.sos/4966833ed02ca234.To reproduce the error please run:
[0m[32msingularity exec  /restricted/projectnb/casa/frank/xqtl_project/xqtl-pipeline/container/singularity/bioinfo.sif python /usr3/graduate/fgrennjr/.sos/4966833ed02ca234/singularity_run_83472.py[0m[91m[0m

[hsun3163]

/opt/conda/bin/python: can't open file '/restricted/projectnb/casa/frank/xqtl_project/ROSMAP_PCC/scripts/tmpdbgt_wz5/singularity_run_83472.py': [Errno 2] No such file or directory

This is not a error by itself per se. SOS will move the py script in the tmp folder to the new destination: /usr3/graduate/fgrennjr/.sos/4966833ed02ca234/singularity_run_83472.py

[netwon123]

so?how to solve it