Errors

[diego-rt]

Hello,

First of all, thanks a lot for the great pipeline.

I'm having a bit of trouble getting stainedGlass to work on my longest sequences however. I'm using the provided condo environments with snakemake and during figure generation I get a variety of different errors, but only for the longest sequences.

This is my command:

mamba activate stainedGlass

samtools faidx ${region_fasta}

snakemake --use-conda --cores ${task.cpus} all make_figures cooler --config sample=${name} fasta=${region_fasta} alnthreads=8 nbatch=16

These are some of the error messages:

[Tue Jan  2 00:47:58 2024]
Finished job 48.
46 of 49 steps (94%) done
Select jobs to execute...

[Tue Jan  2 00:47:58 2024]
localrule cooler:
    input: results/chr2_centromere.2000.10000.strand.cool, results/chr2_centromere.2000.10000.identity.cool, results/chr2_centromere.2000.10000.strand.mcool, results/chr2_centromere.2000.10000.identity.mcool
    jobid: 44
    reason: Input files updated by another job: results/chr2_centromere.2000.10000.strand.mcool, results/chr2_centromere.2000.10000.identity.cool, results/chr2_centromere.2000.10000.identity.mcool, results/chr2_centromere.2000.10000.strand.cool
    resources: tmpdir=/tmp

[Tue Jan  2 00:47:58 2024]
Finished job 44.
47 of 49 steps (96%) done
Error in checkForRemoteErrors(val) : 
  one node produced an error: there is no package called ‘labeling’
Calls: parLapply ... clusterApply -> staticClusterApply -> checkForRemoteErrors
Execution halted
Error: package ‘tools’ does not have a namespace
Error in unserialize(node$con) : error reading from connection
Calls: <Anonymous> ... doTryCatch -> recvData -> recvData.SOCKnode -> unserialize
Execution halted
Error: package ‘tools’ does not have a namespace
[Tue Jan  2 04:16:25 2024]
Error in rule make_R_figures:
    jobid: 43
    input: results/chr2_centromere.2000.10000.bed.gz
    output: results/chr2_centromere.2000.10000_figures/pdfs/chr2_centromere.2000.10000.tri.TRUE__onecolorscale.FALSE__all.pdf, results/chr2_centromere.2000.10000_figures/pngs/chr2_centromere.2000.10000.tri.TRUE__onecolorscale.TRUE__all.png, results/chr2_centromere.2000.10000_figures/pngs/chr2_centromere.2000.10000.tri.TRUE__onecolorscale.FALSE__all.png, results/chr2_centromere.2000.10000_figures/pngs/chr2_centromere.2000.10000.tri.FALSE__onecolorscale.TRUE__all.png, results/chr2_centromere.2000.10000_figures/pngs/chr2_centromere.2000.10000.tri.FALSE__onecolorscale.FALSE__all.png, results/chr2_centromere.2000.10000_figures/pngs/chr2_centromere.2000.10000.facet.all.png
    log: logs/make_R_figures.chr2_centromere.2000.10000.log (check log file(s) for error details)
    conda-env: /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/cb/309e7904e7bbeb60a61c79c1f50d37/.snakemake/conda/5d32f79cd3fb5bb0c333cf2c276a3eaf_
    shell:
        
        Rscript /users/diego.terrones/.cache/snakemake/snakemake/source-cache/runtime-cache/tmpqrqptiq2/file/scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/cb/309e7904e7bbeb60a61c79c1f50d37/workflow/scripts/aln_plot.R             --bed results/chr2_centromere.2000.10000.bed.gz             --threads 16             --prefix chr2_centromere.2000.10000
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-01-01T164206.294919.snakemake.log
Traceback (most recent call last):
  File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/weakref.py", line 667, in _exitfunc
    f()
  File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/tempfile.py", line 820, in _cleanup
    cls._rmtree(name)
  File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/tempfile.py", line 816, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 734, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 667, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 667, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 667, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  [Previous line repeated 8 more times]
  File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 673, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/users/diego.terrones/miniforge3/envs/stainedGlass/lib/python3.9/shutil.py", line 671, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'scripts'

Or this one:

Activating conda environment: .snakemake/conda/284fb86d478e02947ecbd7c10b955f52_
[Tue Jan  2 01:45:06 2024]
localrule all:
    input: results/chr4_centromere.2000.10000.sorted.bam, results/chr4_centromere.2000.10000.bed.gz, results/chr4_centromere.2000.10000.full.tbl.gz
    jobid: 43
    reason: Input files updated by another job: results/chr4_centromere.2000.10000.sorted.bam, results/chr4_centromere.2000.10000.full.tbl.gz, results/chr4_centromere.2000.10000.bed.gz
    resources: tmpdir=/tmp

[Tue Jan  2 01:45:06 2024]
Finished job 43.
42 of 49 steps (86%) done
/scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/d959733dddc863405d2eb9dbb534ae0d_/lib/R/bin/R: line 272: /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/d959733dddc863405d2eb9dbb534ae0d_/lib/R/bin/exec/R: No such file or directory
[Tue Jan  2 01:45:06 2024]
Error in rule make_R_figures:
    jobid: 1
    input: results/chr4_centromere.2000.10000.bed.gz
    output: results/chr4_centromere.2000.10000_figures/pdfs/chr4_centromere.2000.10000.tri.TRUE__onecolorscale.FALSE__all.pdf, results/chr4_centromere.2000.10000_figures/pngs/chr4_centromere.2000.10000.tri.TRUE__onecolorscale.TRUE__all.png, results/chr4_centromere.2000.10000_figures/pngs/chr4_centromere.2000.10000.tri.TRUE__onecolorscale.FALSE__all.png, results/chr4_centromere.2000.10000_figures/pngs/chr4_centromere.2000.10000.tri.FALSE__onecolorscale.TRUE__all.png, results/chr4_centromere.2000.10000_figures/pngs/chr4_centromere.2000.10000.tri.FALSE__onecolorscale.FALSE__all.png, results/chr4_centromere.2000.10000_figures/pngs/chr4_centromere.2000.10000.facet.all.png
    log: logs/make_R_figures.chr4_centromere.2000.10000.log (check log file(s) for error details)
    conda-env: /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/d959733dddc863405d2eb9dbb534ae0d_
    shell:
        
        Rscript /users/diego.terrones/.cache/snakemake/snakemake/source-cache/runtime-cache/tmp1vhq84v1/file/scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/workflow/scripts/aln_plot.R             --bed results/chr4_centromere.2000.10000.bed.gz             --threads 16             --prefix chr4_centromere.2000.10000
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Traceback (most recent call last):
Traceback (most recent call last):
  File "/scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/284fb86d478e02947ecbd7c10b955f52_/bin/cooler", line 7, in <module>
  File "/scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/284fb86d478e02947ecbd7c10b955f52_/bin/cooler", line 7, in <module>
    from cooler.cli import cli
    from cooler.cli import cli
ImportError: cannot import name 'cli' from 'cooler.cli' (unknown location)
ImportError: cannot import name 'cli' from 'cooler.cli' (unknown location)
[Tue Jan  2 01:45:07 2024]
Error in rule cooler_strand:
    jobid: 45
    input: results/chr4_centromere.2000.10000.bed.gz, /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/regions.part_004.fa.fai
    output: results/chr4_centromere.2000.10000.strand.cool
    log: logs/cooler_strand.chr4_centromere.2000.10000.log (check log file(s) for error details)
    conda-env: /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/284fb86d478e02947ecbd7c10b955f52_
    shell:
        
        gunzip -c results/chr4_centromere.2000.10000.bed.gz | tail -n +2             | sed  's/+$/100/g'             | sed  's/-$/50/g'               | cooler cload pairs                 -c1 1 -p1 2 -c2 4 -p2 5                         --field count=8:agg=mean,dtype=float                 --chunksize 50000000000                 /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/regions.part_004.fa.fai:2000                 --zero-based                 - results/chr4_centromere.2000.10000.strand.cool
        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[Tue Jan  2 01:45:07 2024]
Error in rule cooler_identity:
    jobid: 46
    input: results/chr4_centromere.2000.10000.bed.gz, /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/regions.part_004.fa.fai
    output: results/chr4_centromere.2000.10000.identity.cool
    log: logs/cooler_identity.chr4_centromere.2000.10000.log (check log file(s) for error details)
    conda-env: /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/.snakemake/conda/284fb86d478e02947ecbd7c10b955f52_
    shell:
        
        gunzip -c results/chr4_centromere.2000.10000.bed.gz | tail -n +2               | cooler cload pairs                 -c1 1 -p1 2 -c2 4 -p2 5                         --field count=7:agg=mean,dtype=float                 --chunksize 50000000000                 /scratch-cbe/users/diego.terrones/5_AxolotlT2T/6_centromereAnalysis/1_Ambystoma-mexicanum/1_StainedGlass/25/abdb34cc958c5259e797ff05e5cab3/regions.part_004.fa.fai:2000                 --zero-based                 - results/chr4_centromere.2000.10000.identity.cool

        
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-01-01T172106.733496.snakemake.log

Weirdly, the shorter sequences work just fine, so I'm a bit confused. Any ideas?