aMeta runtest.sh fails at MAKING AUTHENTICATION AND VALIDATION PLOTS

Hi, I installed aMeta from scratch using a newly installed version of micromamba, and I am not able to get runtests.sh to complete successfully. See below for the full output of the step that failed.

The procedure I am following is to take the output of the shell: segment from the output, copying the commands to a file, then running each command to determine which of several commands is failing and why from the visible error message(s).

The first errors were because Rscript was failing because of missing libraries, gridBase and lifecycle. I ran the R in the aMeta environment to install those, and that seemed to get past that.

> install.package('lifecycle')
> install.packages('gridBase')

The second was because some directories were not being created prior to copy commands with them as the target, which I think I fixed by adding mkdir commands to precreate them to authentic.smk

    shell:
        "mkdir -p logs/AUTHENTICATION_PLOTS;"
        "mkdir -p results/AUTHENTICATION_PLOTS_PDF;"
        "mkdir -p results/AUTHENTICATION_PLOTS_PNG;"
        "Rscript {params.exe} {wildcards.taxid} {wildcards.sample}.trimmed.rma6 {input.dir}/;"
        "cp -v {output.pdf_plot} {output.pdf};"
        "cp -v {output.png_plot} {output.png};"

in aMeta/workflow/rules.

That got me to the point where the commands that Snakemake is executing are (with lines

mkdir -p logs/AUTHENTICATION_PLOTS;
mkdir -p results/AUTHENTICATION_PLOTS_PDF;
mkdir -p results/AUTHENTICATION_PLOTS_PNG;
Rscript /tmp/bfauber/aMeta/workflow/scripts/authentic.R 632 foo.trimmed.rma6 results/AUTHENTICATION/foo/632/;
cp -v results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf \
    results/AUTHENTICATION_PLOTS_PDF/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf;
cp -v results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.png \
    results/AUTHENTICATION_PLOTS_PNG/authentic_Sample_foo.trimmed.rma6_TaxID_632.png;

and when I copy those to a separate file, ,w, and run that, I get no errors.

$ source ,w
pdf output: results/AUTHENTICATION/foo/632//authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf
Read 16 items
null device 
          1 
WARNING: The convert command is deprecated in IMv7, use "magick"

'results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf' -> 'results/AUTHENTICATION_PLOTS_PDF/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf'
'results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.png' -> 'results/AUTHENTICATION_PLOTS_PNG/authentic_Sample_foo.trimmed.rma6_TaxID_632.png'

But it doesn't seem to work from Snakemake.

Any help would be appreciated.

Here is the command and edited output from running the runtest.sh script.

$ bash runtest.sh -j 1
Conda version 24.7.1: using conda frontend conda
Running workflow...
snakemake --conda-frontend conda --use-conda --show-failed-logs --conda-cleanup-pkgs cache -s ../workflow/Snakefile -j 1

. . . .  output elided . . . .

[Tue Jan  7 12:35:43 2025]
Job 40: Authentication_Plots: MAKING AUTHENTICATION AND VALIDATION PLOTS
Reason: Missing output files: results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf; Updated input files: results/AUTHENTICATION/foo/632

Activating conda environment: .snakemake/conda/c69b3c6f8bfc081337fafbe335c2fe63_
Read 16 items
null device 
          1 
sh: line 1: convert: command not found
'results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf' -> 'results/AUTHENTICATION_PLOTS_PDF/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf'
cp: cannot stat 'results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.png': No such file or directory
[Tue Jan  7 12:35:44 2025]
Error in rule Authentication_Plots:
    jobid: 40
    input: results/AUTHENTICATION/foo/632, results/AUTHENTICATION/foo/632/node_list.txt, results/AUTHENTICATION/foo/632/read_length.txt, results/AUTHENTICATION/foo/632/PMDscores.txt, results/AUTHENTICATION/foo/632/breadth_of_coverage
    output: results/AUTHENTICATION_PLOTS_PDF/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf, results/AUTHENTICATION_PLOTS_PNG/authentic_Sample_foo.trimmed.rma6_TaxID_632.png, results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf, results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.png
    log: logs/AUTHENTICATION_PLOTS/foo_632.log (check log file(s) for error details)
    conda-env: /tmp/bfauber/aMeta/.test/.snakemake/conda/c69b3c6f8bfc081337fafbe335c2fe63_
    shell:
        mkdir -p logs/AUTHENTICATION_PLOTS;mkdir -p results/AUTHENTICATION_PLOTS_PDF;mkdir -p results/AUTHENTICATION_PLOTS_PNG;Rscript /tmp/bfauber/aMeta/workflow/scripts/authentic.R 632 foo.trimmed.rma6 results/AUTHENTICATION/foo/632/;cp -v results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf results/AUTHENTICATION_PLOTS_PDF/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf;cp -v results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.png results/AUTHENTICATION_PLOTS_PNG/authentic_Sample_foo.trimmed.rma6_TaxID_632.png;
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Logfile logs/AUTHENTICATION_PLOTS/foo_632.log not found.

Removing output files of failed job Authentication_Plots since they might be corrupted:
results/AUTHENTICATION_PLOTS_PDF/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf, results/AUTHENTICATION/foo/632/authentic_Sample_foo.trimmed.rma6_TaxID_632.pdf
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2025-01-07T123541.359079.snakemake.log
ERROR: Workflow test failed!

Jan 07 '25 18:01 justbennet

Hi,

the convert command is part of imagemagick. However, that dependency seems to be missing in the environment file for the Authentication_Plots rule in rules/authentic.smk. What happens if you add imagemagick to the dependency list in envs/malt.yaml?

Jan 20 '25 09:01 percyfal

I just noticed that error as well when running the new aMeta version. We need to add imagemagick to aMeta/workflow/envs/malt.yaml.

Concretely for you @justbennet, it means you need to open aMeta/workflow/envs/malt.yaml which should look like this (and add this last line I added)

channels:

conda-forge
bioconda

dependencies:

hops
pmdtools
samtools
seqtk
megan
imagemagick

Feb 03 '25 15:02 ZoePochon