Failed to extract ref_id from results/AUTHENTICATION/

[mudymudy]

Hi everyone,

I've been running the aMeta pipeline for over a week (I have many samples) but it got stuck in a step for days. I cancelled the run and then I run it again with --rerun-incomplete.

This is a snippet of the error:

Failed to extract ref_id from results/AUTHENTICATION/sample/40137/MaltExtract_output/default/readDist/sample.trimmed.rma6_additionalNodeEntries.txt; returning taxid 40137 Failed to extract ref_id from results/AUTHENTICATION/sample/1063/MaltExtract_output/default/readDist/sample.trimmed.rma6_additionalNodeEntries.txt; returning taxid 1063 Failed to extract ref_id from results/AUTHENTICATION/sample/40137/MaltExtract_output/default/readDist/sample.trimmed.rma6_additionalNodeEntries.txt; returning taxid 40137 Failed to extract ref_id from results/AUTHENTICATION/sample/1063/MaltExtract_output/default/readDist/sample.trimmed.rma6_additionalNodeEntries.txt; returning taxid 1063 Failed to extract ref_id from results/AUTHENTICATION/sample/40137/MaltExtract_output/default/readDist/sample.trimmed.rma6_additionalNodeEntries.txt; returning taxid 40137 Failed to extract ref_id from results/AUTHENTICATION/sample/1063/MaltExtract_output/default/readDist/sample.trimmed.rma6_additionalNodeEntries.txt; returning taxid 1063 Failed to extract ref_id from results/AUTHENTICATION/sample/40137/MaltExtract_output/default/readDist/sample.trimmed.rma6_additionalNodeEntries.txt; returning taxid 40137 Failed to extract ref_id from results/AUTHENTICATION/sample/1063/MaltExtract_output/default/readDist/sample.trimmed.rma6_additionalNodeEntries.txt; returning taxid 1063 Using shell: /bin/bash Provided cores: 40 Rules claiming more threads will be scaled down. Singularity containers: ignored Job counts: count jobs 3933 Authentication_Plots 3933 Authentication_Score 3933 Breadth_Of_Coverage 3933 Deamination 3933 PMD_scores 1 Plot_Authentication_Score 3933 Read_Length_Distribution 1 Samtools_Faidx 63 aggregate 1 all 23664 Select jobs to execute...

[Thu Jun 27 11:01:50 2024] Job 927: Samtools_Faidx: INDEXING MALT FASTA DATABASE FOR BREADTH_OF_COVERAGE SEQUENCE RETRIEVAL

Activating conda environment: /home/user/aMeta/.snakemake/conda/d6b2d8fd2b7148be9872ee0d2fc560c1 /bin/bash: /home/user/aMeta_db/malt-index/library.fna.fai_Samtools_Faidx.log: Permission denied [Thu Jun 27 11:01:54 2024] Error in rule Samtools_Faidx: jobid: 927 output: /home/user/aMeta_db/malt-index/library.fna.fai log: /home/user/aMeta_db/malt-index/library.fna.fai_Samtools_Faidx.log (check log file(s) for error message) conda-env: /home/user/aMeta/.snakemake/conda/d6b2d8fd2b7148be9872ee0d2fc560c1 shell: samtools faidx /home/user/aMeta_db/malt-index/library.fna 2> /home/user/aMeta_db/malt-index/library.fna.fai_Samtools_Faidx.log (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

exception calling callback for <Future at 0x7ef94758ac80 state=finished raised RuleException> Traceback (most recent call last): File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/executors/init.py", line 2330, in run_wrapper run( File "/home/user/aMeta/workflow/rules/authentic.smk", line 259, in __rule_Samtools_Faidx message: File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/shell.py", line 231, in new raise sp.CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'source /home/user/miniforge3/envs/aMeta/bin/activate '/home/user/aMeta/.snakemake/conda/d6b2d8fd2b7148be9872ee0d2fc560c1'; set -euo pipefail; samtools faidx /home/user/aMeta_db/malt-index/library.fna 2> /home/user/aMeta_db/malt-index/library.fna.fai_Samtools_Faidx.log' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/executors/init.py", line 569, in _callback raise ex File "/home/user/miniforge3/envs/aMeta/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/executors/init.py", line 555, in cached_or_run run_func(*args) File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/executors/init.py", line 2362, in run_wrapper raise RuleException( snakemake.exceptions.RuleException: CalledProcessError in line 135 of /home/user/aMeta/workflow/rules/authentic.smk: Command 'source /home/user/miniforge3/envs/aMeta/bin/activate '/home/user/aMeta/.snakemake/conda/d6b2d8fd2b7148be9872ee0d2fc560c1'; set -euo pipefail; samtools faidx /home/user/aMeta_db/malt-index/library.fna 2> /home/user/aMeta_db/malt-index/library.fna.fai_Samtools_Faidx.log' returned non-zero exit status 1. File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/executors/init.py", line 2330, in run_wrapper File "/home/user/aMeta/workflow/rules/authentic.smk", line 135, in __rule_Samtools_Faidx

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/io.py", line 644, in touch lutime(self.file, times) File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/io.py", line 67, in lutime os.utime(f, times, follow_symlinks=False) FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/io.py", line 658, in touch_or_create self.touch() File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/io.py", line 647, in touch raise MissingOutputException( snakemake.exceptions.MissingOutputException: Job Output file /home/user/aMeta_db/malt-index/library.fna.fai_Samtools_Faidx.log of rule Samtools_Faidx shall be touched but does not exist. completed successfully, but some output files are missing.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/user/miniforge3/envs/aMeta/lib/python3.10/concurrent/futures/_base.py", line 342, in _invoke_callbacks callback(self) File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/executors/init.py", line 581, in _callback error_callback(job) File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/scheduler.py", line 579, in _error self._handle_error(job) File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/scheduler.py", line 588, in _handle_error self.get_executor(job).handle_job_error(job) File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/executors/init.py", line 587, in handle_job_error super().handle_job_error(job) File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/executors/init.py", line 264, in handle_job_error job.postprocess( File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/jobs.py", line 980, in postprocess self.dag.handle_log(self) File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/dag.py", line 625, in handle_log f.touch_or_create() File "/home/user/miniforge3/envs/aMeta/lib/python3.10/site-packages/snakemake/io.py", line 670, in touch_or_create with open(file, "w") as f: PermissionError: [Errno 13] Permission denied: '/home/user/aMeta_db/malt-index/library.fna.fai_Samtools_Faidx.log'

Here it says:

/bin/bash: /home/user/aMeta_db/malt-index/library.fna.fai_Samtools_Faidx.log: Permission denied

But every other file in the aMeta db has the same permission level, so I think maybe the file is just not being generated somehow?

I still have many edit distances (both ancient and default) in the AUTHENTICATION folder but it would be great if the pipeline can finish properly!

Thank you in advance!