OSError: [Errno 39] Directory not empty: 'envs'

[shashwatsahay]

Snakemake version 7.32.4 or 7.32.3

Describe the bug The pipeline runs fine but at the end gives this error and ends the run

Logs

Building DAG of jobs...
Nothing to be done (all requested files are present and up to date).
Complete log: .snakemake/log/2024-03-02T121659.175791.snakemake.log
Traceback (most recent call last):
  File "/home/sahay/miniconda3/envs/panel-umi-dedup-full/lib/python3.10/weakref.py", line 667, in _exitfunc
    f()
  File "/home/sahay/miniconda3/envs/panel-umi-dedup-full/lib/python3.10/weakref.py", line 591, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/home/sahay/miniconda3/envs/panel-umi-dedup-full/lib/python3.10/tempfile.py", line 859, in _cleanup
    cls._rmtree(name, ignore_errors=ignore_errors)
  File "/home/sahay/miniconda3/envs/panel-umi-dedup-full/lib/python3.10/tempfile.py", line 855, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "/home/sahay/miniconda3/envs/panel-umi-dedup-full/lib/python3.10/shutil.py", line 725, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/sahay/miniconda3/envs/panel-umi-dedup-full/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/sahay/miniconda3/envs/panel-umi-dedup-full/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/home/sahay/miniconda3/envs/panel-umi-dedup-full/lib/python3.10/shutil.py", line 658, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  [Previous line repeated 5 more times]
  File "/home/sahay/miniconda3/envs/panel-umi-dedup-full/lib/python3.10/shutil.py", line 664, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/home/sahay/miniconda3/envs/panel-umi-dedup-full/lib/python3.10/shutil.py", line 662, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'envs'

Minimal example snakemake --slurm -j 10 --configfile config/S7.yaml --profile profile --snakefile /workflow/Snakefile

[johanneskoester]

Looks a bit like a not Snakemake related race condition in your file system. Nevertheless, I would try with latest Snakemake.

[shashwatsahay]

Hi @johanneskoester

I am restricted currently to the snakemake 7 due to compatibility issues with our Workflow execution service

Nonetheless I will test case a single run with snakemake 8.

Also I don't know if this is related to the problem, but this only comes up if I run the pipeline when it didn't sucessfully complete in a previous run, for new runs this doesn't show up, neither does it show when encapsulating the pipeline inside a singularity container.

[GoodingJamie]

Hi both, I'm having exactly the same issue when using the hadd wrapper from https://github.com/snakemake/snakemake-wrappers/tree/master/phys/root/hadd in Snakemake 7.

The error is identical to above but with OSError: [Errno 39] Directory not empty: 'hadd' as the final line) in Snakemake 7 but not in Snakemake 8.

[angrybee]

I have tried to do a minimal example.

1. Run with v8.10.8 without any complains
2. Re-run a. v8.10.8 has OSError (error.log) b. v8.7.0 has no OSError

Edit: upgrade of wrapper to v3.9.0 shows same problem Edit2:

mamba --version
mamba 1.1.0
conda 22.9.0

Environment v8.7.0: snakemake_8.7.0.txt snakemake_8.7.0_links.txt

Environment v8.10.8: snakemake_8.10.8.txt snakemake_8.10.8_links.txt

Snakefile:

#!/bin/python
rule get_fastq_pe_gz:
    output:
        "{accession}_1.fastq.gz",
        "{accession}_2.fastq.gz",
    log:
        "{accession}.log",
    params:
        extra="--skip-technical -x"
    threads: 6
    wrapper:
        "v3.3.6/bio/sra-tools/fasterq-dump"

rule all:
    input:
        expand(
            "{accession}_{end}.fastq.gz",
            accession=["SRR18057178"],
            end=[1, 2],
        ),

run.sh:

#!/bin/bash
conda config --set channel_priority strict

snakemake \
	--use-conda \
	--conda-frontend mamba \
	--cores 1 \
	--keep-going \
	--rerun-incomplete \
	--default-resources "tmpdir='/tmp'" \
	-p \
	all

[angrybee]

No re-run issues with a very minimalistic snakefile and without wrapper.

#!/bin/python
rule get_fastq_pe_gz:
    output:
        "{accession}_1.fastq.gz",
        "{accession}_2.fastq.gz",
    log:
        "{accession}.log",
    shell:
        """
        touch {output}
        """

rule all:
    input:
        expand(
            "{accession}_{end}.fastq.gz",
            accession=["touch_me"],
            end=[1, 2],
        ),

[johanneskoester]

This is fixed now in #2859