(ERR): "Arabidopsis_simulated.plastome/seed/embplant_pt.index" does not exist or is not a Bowtie 2 index

[tallnuttrbgv]

Hi,

I have installed using a manual method - git clone etc. But it seems to fail on test data building the bowtie index.

Thanks,

GetOrganelle v1.7.5.3

get_organelle_from_reads.py assembles organelle genomes from genome skimming data. Find updates in https://github.com/Kinggerm/GetOrganelle and see README.md for more information.

Python 3.10.0 (default, Nov 16 2021, 09:41:50) [GCC 8.4.1 20200928 (Red Hat 8.4.1-1)] PLATFORM: Linux gadi-login-03.gadi.nci.org.au 4.18.0-348.2.1.el8.nci.x86_64 #1 SMP Fri Nov 26 03:20:41 UTC 2021 x86_64 x86_64 PYTHON LIBS: GetOrganelleLib 1.7.5.3; numpy 1.21.4; sympy 1.9; scipy 1.7.2; psutil 5.9.0 DEPENDENCIES: Bowtie2 2.3.5.1; /g/data/nm31/bin/GetOrganelle/GetOrganelleDep/linux/SPAdes/bin/spades.py:13: DeprecationWarning: The distutils package is deprecated and slated for remoal in Python 3.12. Use setuptools or check PEP 632 for potential alternaties from distutils import dir_util SPAdes 3.13.0; Blast 2.9.0 GETORG_PATH=/home/554/ta0341/.GetOrganelle SEED DB: embplant_pt 0.0.1; embplant_mt 0.0.1 LABEL DB: embplant_pt 0.0.1; embplant_mt 0.0.1 WORKING DIR: /g/data/nm31/d/r3.22_paftol_validation/getorgtest /g/data/nm31/bin/GetOrganelle/get_organelle_from_reads.py -1 Arabidopsis_simulated.1.fq.gz -2 Arabidopsis_simulated.2.fq.gz -t 1 -o Arabidopsis_simulated.plastome -F embplant_pt -R 10 --overwrite

2022-02-22 15:37:02,907 - INFO: Pre-reading fastq ... 2022-02-22 15:37:02,907 - INFO: Estimating reads to use ... (to use all reads, set '--reduce-reads-for-coverage inf --max-reads inf') 2022-02-22 15:37:03,028 - INFO: Estimating reads to use finished. 2022-02-22 15:37:03,029 - INFO: Unzipping reads file: Arabidopsis_simulated.1.fq.gz (8796915 bytes) 2022-02-22 15:37:07,535 - INFO: Unzipping reads file: Arabidopsis_simulated.2.fq.gz (9067061 bytes) 2022-02-22 15:37:12,807 - INFO: Counting read qualities ... 2022-02-22 15:37:12,959 - INFO: Identified quality encoding format = Illumina 1.8+ 2022-02-22 15:37:12,959 - INFO: Phred offset = 33 2022-02-22 15:37:12,960 - INFO: Trimming bases with qualities (0.00%): 33..33 ! 2022-02-22 15:37:13,012 - INFO: Mean error rate = 0.0019 2022-02-22 15:37:13,013 - INFO: Counting read lengths ... 2022-02-22 15:37:13,181 - INFO: Mean = 150.0 bp, maximum = 150 bp. 2022-02-22 15:37:13,182 - INFO: Reads used = 91563+91563 2022-02-22 15:37:13,182 - INFO: Pre-reading fastq finished.

2022-02-22 15:37:13,182 - INFO: Making seed reads ... 2022-02-22 15:37:18,147 - INFO: Making seed - bowtie2 index ... 2022-02-22 15:37:18,212 - INFO: Making seed - bowtie2 index finished. 2022-02-22 15:37:18,213 - INFO: Mapping reads to seed bowtie2 index ... 2022-02-22 15:37:18,316 - ERROR: (ERR): "Arabidopsis_simulated.plastome/seed/embplant_pt.index" does not exist or is not a Bowtie 2 index Exiting now ...

2022-02-22 15:37:18,316 - ERROR: Traceback (most recent call last): File "/g/data/nm31/bin/GetOrganelle/get_organelle_from_reads.py", line 3941, in main seed_fq, seed_sam, new_seed_f = making_seed_reads_using_mapping( File "/g/data/nm31/bin/GetOrganelle/get_organelle_from_reads.py", line 3040, in making_seed_reads_using_mapping map_with_bowtie2(seed_file=seed_file, original_fq_files=original_fq_files, File "/g/data/nm31/bin/GetOrganelle/GetOrganelleLib/pipe_control_func.py", line 399, in map_with_bowtie2 raise Exception("") Exception

Total cost 22.27 s For trouble-shooting, please Firstly, check https://github.com/Kinggerm/GetOrganelle/wiki/FAQ Secondly, check if there are open/closed issues related at https://github.com/Kinggerm/GetOrganelle/issues If your problem was still not solved, please open an issue at https://github.com/Kinggerm/GetOrganelle/issues please provide the get_org.log.txt and the assembly graph (can be *.png to protect your data privacy) if possible!

[jaktykusuma]

Hi,

Sorry to interrupt and add a few problems in this thread. You might want to download the database first. Put it in your main directory https://github.com/Kinggerm/GetOrganelle/wiki/Initialization

I had the same trouble but succeed after downloading it.

However, I notice the same error I made with your log. If you see the dependencies, it's deprecated. I hope the authors could help us fix this problem.

See my log file. Something wrong with SPAdes.

get_org.log.txt

Thank you.

[Kinggerm]

jaktykusuma

Your error was different from the current thread. The deprecated dependency issue is currently a harmless warning, not an error.

The failure of running SPAdes in your case was caused by the space in your working directory, specifically, "IRD Works". Besides, please to 1.7.5+, which not only has better instant feedback info in the space-in-working-directory case but also has essential bugs fixed.

[Kinggerm]

Hi,

I have installed using a manual method - git clone etc. But it seems to fail on test data building the bowtie index.

Thanks,

GetOrganelle v1.7.5.3

get_organelle_from_reads.py assembles organelle genomes from genome skimming data. Find updates in https://github.com/Kinggerm/GetOrganelle and see README.md for more information.

Python 3.10.0 (default, Nov 16 2021, 09:41:50) [GCC 8.4.1 20200928 (Red Hat 8.4.1-1)] PLATFORM: Linux gadi-login-03.gadi.nci.org.au 4.18.0-348.2.1.el8.nci.x86_64 #1 SMP Fri Nov 26 03:20:41 UTC 2021 x86_64 x86_64 PYTHON LIBS: GetOrganelleLib 1.7.5.3; numpy 1.21.4; sympy 1.9; scipy 1.7.2; psutil 5.9.0 DEPENDENCIES: Bowtie2 2.3.5.1; /g/data/nm31/bin/GetOrganelle/GetOrganelleDep/linux/SPAdes/bin/spades.py:13: DeprecationWarning: The distutils package is deprecated and slated for remoal in Python 3.12. Use setuptools or check PEP 632 for potential alternaties from distutils import dir_util SPAdes 3.13.0; Blast 2.9.0 GETORG_PATH=/home/554/ta0341/.GetOrganelle SEED DB: embplant_pt 0.0.1; embplant_mt 0.0.1 LABEL DB: embplant_pt 0.0.1; embplant_mt 0.0.1 WORKING DIR: /g/data/nm31/d/r3.22_paftol_validation/getorgtest /g/data/nm31/bin/GetOrganelle/get_organelle_from_reads.py -1 Arabidopsis_simulated.1.fq.gz -2 Arabidopsis_simulated.2.fq.gz -t 1 -o Arabidopsis_simulated.plastome -F embplant_pt -R 10 --overwrite

2022-02-22 15:37:02,907 - INFO: Pre-reading fastq ... 2022-02-22 15:37:02,907 - INFO: Estimating reads to use ... (to use all reads, set '--reduce-reads-for-coverage inf --max-reads inf') 2022-02-22 15:37:03,028 - INFO: Estimating reads to use finished. 2022-02-22 15:37:03,029 - INFO: Unzipping reads file: Arabidopsis_simulated.1.fq.gz (8796915 bytes) 2022-02-22 15:37:07,535 - INFO: Unzipping reads file: Arabidopsis_simulated.2.fq.gz (9067061 bytes) 2022-02-22 15:37:12,807 - INFO: Counting read qualities ... 2022-02-22 15:37:12,959 - INFO: Identified quality encoding format = Illumina 1.8+ 2022-02-22 15:37:12,959 - INFO: Phred offset = 33 2022-02-22 15:37:12,960 - INFO: Trimming bases with qualities (0.00%): 33..33 ! 2022-02-22 15:37:13,012 - INFO: Mean error rate = 0.0019 2022-02-22 15:37:13,013 - INFO: Counting read lengths ... 2022-02-22 15:37:13,181 - INFO: Mean = 150.0 bp, maximum = 150 bp. 2022-02-22 15:37:13,182 - INFO: Reads used = 91563+91563 2022-02-22 15:37:13,182 - INFO: Pre-reading fastq finished.

2022-02-22 15:37:13,182 - INFO: Making seed reads ... 2022-02-22 15:37:18,147 - INFO: Making seed - bowtie2 index ... 2022-02-22 15:37:18,212 - INFO: Making seed - bowtie2 index finished. 2022-02-22 15:37:18,213 - INFO: Mapping reads to seed bowtie2 index ... 2022-02-22 15:37:18,316 - ERROR: (ERR): "Arabidopsis_simulated.plastome/seed/embplant_pt.index" does not exist or is not a Bowtie 2 index Exiting now ...

2022-02-22 15:37:18,316 - ERROR: Traceback (most recent call last): File "/g/data/nm31/bin/GetOrganelle/get_organelle_from_reads.py", line 3941, in main seed_fq, seed_sam, new_seed_f = making_seed_reads_using_mapping( File "/g/data/nm31/bin/GetOrganelle/get_organelle_from_reads.py", line 3040, in making_seed_reads_using_mapping map_with_bowtie2(seed_file=seed_file, original_fq_files=original_fq_files, File "/g/data/nm31/bin/GetOrganelle/GetOrganelleLib/pipe_control_func.py", line 399, in map_with_bowtie2 raise Exception("") Exception

Total cost 22.27 s For trouble-shooting, please Firstly, check https://github.com/Kinggerm/GetOrganelle/wiki/FAQ Secondly, check if there are open/closed issues related at https://github.com/Kinggerm/GetOrganelle/issues If your problem was still not solved, please open an issue at https://github.com/Kinggerm/GetOrganelle/issues please provide the get_org.log.txt and the assembly graph (can be *.png to protect your data privacy) if possible!

Could you please

1. run ls -lah Arabidopsis_simulated.plastome/seed/embplant_pt.index to list the files
2. run bowtie2-build -h to see the reaction.
3. rerun the command with "--verbose" added and attach the new log file here

Thanks!

[tallnuttrbgv]

ls -lah Arabidopsis_simulated.plastome/seed/embplant_pt.index ls: cannot access 'Arabidopsis_simulated.plastome/seed/embplant_pt.index': No such file or directory

ls -lah Arabidopsis_simulated.plastome/seed/ total 15M drwxr-sr-x 2 ta0341 nm31 33K Feb 22 15:44 . drwxr-sr-x 3 ta0341 nm31 33K Feb 22 15:44 .. -rw-r--r-- 1 ta0341 nm31 15M Feb 22 15:44 embplant_pt.fasta

bowtie2-build -h === ERROR === The use of the #!/usr/bin/env python interpreter line in python scripts has been deprecated.

Please modify this script: /g/data/nm31/bin/GetOrganelle/GetOrganelleDep/linux/bowtie2/bowtie2-build

To use either #!/usr/bin/env python3 or #!/usr/bin/env python2 depending on which version of python you require Alternatively, if you are unable to modify this script You can load the python2-as-python or python3-as-python modules depending on which version of python you require

I fixed the interpreter line in bowtie2-build then got the error below. Verbose log also attached

get_organelle_from_reads.py -1 Arabidopsis_simulated.1.fq.gz -2 Arabidopsis_simulated.2.fq.gz -t 1 -o Arabidopsis_simulated.plastome -F embplant_pt -R 10 --overwrite
GetOrganelle v1.7.5.3

get_organelle_from_reads.py assembles organelle genomes from genome skimming data. Find updates in https://github.com/Kinggerm/GetOrganelle and see README.md for more information.

Python 3.10.0 (default, Nov 16 2021, 09:41:50) [GCC 8.4.1 20200928 (Red Hat 8.4.1-1)] PLATFORM: Linux gadi-login-06.gadi.nci.org.au 4.18.0-348.2.1.el8.nci.x86_64 #1 SMP Fri Nov 26 03:20:41 UTC 2021 x86_64 x86_64 PYTHON LIBS: GetOrganelleLib 1.7.5.3; numpy 1.21.4; sympy 1.9; scipy 1.7.2; psutil 5.9.0 DEPENDENCIES: Bowtie2 2.3.5.1; /g/data/nm31/bin/GetOrganelle/GetOrganelleDep/linux/SPAdes/bin/spades.py:13: DeprecationWarning: The distutils package is deprecated and slated for remoal in Python 3.12. Use setuptools or check PEP 632 for potential alternaties from distutils import dir_util SPAdes 3.13.0; Blast 2.9.0 GETORG_PATH=/home/554/ta0341/.GetOrganelle SEED DB: embplant_pt 0.0.1; embplant_mt 0.0.1 LABEL DB: embplant_pt 0.0.1; embplant_mt 0.0.1 WORKING DIR: /g/data/nm31/d/r3.22_paftol_validation/getorgtest /g/data/nm31/bin/GetOrganelle/get_organelle_from_reads.py -1 Arabidopsis_simulated.1.fq.gz -2 Arabidopsis_simulated.2.fq.gz -t 1 -o Arabidopsis_simulated.plastome -F embplant_pt -R 10 --overwrite

2022-02-23 10:50:16,977 - INFO: Pre-reading fastq ... 2022-02-23 10:50:16,977 - INFO: Estimating reads to use ... (to use all reads, set '--reduce-reads-for-coverage inf --max-reads inf') 2022-02-23 10:50:17,177 - INFO: Estimating reads to use finished. 2022-02-23 10:50:17,177 - INFO: Unzipping reads file: Arabidopsis_simulated.1.fq.gz (8796915 bytes) 2022-02-23 10:50:17,513 - INFO: Unzipping reads file: Arabidopsis_simulated.2.fq.gz (9067061 bytes) 2022-02-23 10:50:20,697 - INFO: Counting read qualities ... 2022-02-23 10:50:20,851 - INFO: Identified quality encoding format = Illumina 1.8+ 2022-02-23 10:50:20,851 - INFO: Phred offset = 33 2022-02-23 10:50:20,852 - INFO: Trimming bases with qualities (0.00%): 33..33 ! 2022-02-23 10:50:20,901 - INFO: Mean error rate = 0.0019 2022-02-23 10:50:20,902 - INFO: Counting read lengths ... 2022-02-23 10:50:21,068 - INFO: Mean = 150.0 bp, maximum = 150 bp. 2022-02-23 10:50:21,068 - INFO: Reads used = 91563+91563 2022-02-23 10:50:21,068 - INFO: Pre-reading fastq finished.

2022-02-23 10:50:21,068 - INFO: Making seed reads ... 2022-02-23 10:50:24,278 - INFO: Making seed - bowtie2 index ... 2022-02-23 10:50:33,840 - INFO: Making seed - bowtie2 index finished. 2022-02-23 10:50:33,840 - INFO: Mapping reads to seed bowtie2 index ... 2022-02-23 10:50:42,532 - INFO: Mapping finished. 2022-02-23 10:50:42,534 - INFO: Seed reads made: Arabidopsis_simulated.plastome/seed/embplant_pt.initial.fq (14144302 bytes) 2022-02-23 10:50:42,535 - INFO: Making seed reads finished.

2022-02-23 10:50:42,535 - INFO: Checking seed reads and parameters ... 2022-02-23 10:50:42,535 - INFO: The automatically-estimated parameter(s) do not ensure the best choice(s). 2022-02-23 10:50:42,535 - INFO: If the result graph is not a circular organelle genome, 2022-02-23 10:50:42,535 - INFO: you could adjust the value(s) of '-w'/'-R' for another new run. 2022-02-23 10:50:45,524 - INFO: Pre-assembling mapped reads ... 2022-02-23 10:50:47,545 - INFO: Retrying with more reads .. 2022-02-23 10:51:06,399 - WARNING: Pre-assembling failed. The estimations for embplant_pt-hitting base-coverage and word size may be misleading. 2022-02-23 10:51:07,664 - INFO: Estimated embplant_pt-hitting base-coverage = 52.85 2022-02-23 10:51:07,876 - INFO: Estimated word size(s): 98 2022-02-23 10:51:07,877 - INFO: Setting '-w 98' 2022-02-23 10:51:07,877 - INFO: Setting '--max-extending-len inf' 2022-02-23 10:51:07,958 - INFO: Checking seed reads and parameters finished.

2022-02-23 10:51:07,958 - INFO: Making read index ... 2022-02-23 10:51:09,003 - INFO: Mem 0.324 G, 178623 candidates in all 183126 reads 2022-02-23 10:51:09,003 - INFO: Pre-grouping reads ... 2022-02-23 10:51:09,004 - INFO: Setting '--pre-w 98' 2022-02-23 10:51:09,030 - INFO: Mem 0.324 G, 4074/4074 used/duplicated 2022-02-23 10:51:09,287 - INFO: Mem 0.324 G, 517 groups made. 2022-02-23 10:51:09,298 - INFO: Making read index finished.

2022-02-23 10:51:09,298 - INFO: Extending ... 2022-02-23 10:51:09,298 - INFO: Adding initial words ... 2022-02-23 10:51:10,821 - INFO: AW 1113742 2022-02-23 10:51:12,411 - INFO: Round 1: 178623/178623 AI 40378 AW 1126044 Mem 0.437 2022-02-23 10:51:13,216 - INFO: Round 2: 178623/178623 AI 40411 AW 1126346 Mem 0.437 2022-02-23 10:51:14,071 - INFO: Round 3: 178623/178623 AI 40411 AW 1126346 Mem 0.437 2022-02-23 10:51:14,072 - INFO: No more reads found and terminated ... 2022-02-23 10:51:14,782 - INFO: Extending finished.

2022-02-23 10:51:14,795 - INFO: Separating extended fastq file ... 2022-02-23 10:51:15,137 - INFO: Setting '-k 21,55,85,115' 2022-02-23 10:51:15,137 - INFO: Assembling using SPAdes ... 2022-02-23 10:51:15,152 - INFO: /g/data/nm31/bin/GetOrganelle/GetOrganelleDep/linux/SPAdes/bin/spades.py -t 1 --phred-offset 33 -1 Arabidopsis_simulated.plastome/extended_1_paired.fq -2 Arabidopsis_simulated.plastome/extended_2_paired.fq --s1 Arabidopsis_simulated.plastome/extended_1_unpaired.fq --s2 Arabidopsis_simulated.plastome/extended_2_unpaired.fq -k 21,55,85,115 -o Arabidopsis_simulated.plastome/extended_spades 2022-02-23 10:51:15,714 - WARNING: Assembling exited halfway.

2022-02-23 10:51:17,441 - ERROR: No valid assembly graph found!

get_org.log.txt

[tallnuttrbgv]

I also checked that all the other python scripts were #!/usr/bin/env python3, as is required for my system.

[Kinggerm]

I would try removing SPAdes under GetOrganelleDep

rm -r /g/data/nm31/bin/GetOrganelle/GetOrganelleDep/linux/SPAdes/

Then install the latest SPAdes using apt install, or conda, or from the source.

Let me know your updates.

[Kinggerm]

BTW, no matter if the latest SPAdes could fix your issue on the Gadi environment or not, the GetOrganelleDep needs an update. I will leave this issue open until an update.

[tallnuttrbgv]

I deleted the dependency version of spades and use my (working) system version. Get the same error - see attached log. Thanks get_org.log.txt .

[Kinggerm]

What is the result of spades.py --test?

[tallnuttrbgv]

ah yes, spades problem..

spades.py --test

== Warning == No assembly mode was specified! If you intend to assemble high-coverage multi-cell/isolate data, use '--isolate' option.

Command line: /g/data/nm31/bin/SPAdes-3.15.2-Linux/bin/spades.py --test

System information: SPAdes version: 3.15.2 Python version: 3.10.0 OS: Linux-4.18.0-348.2.1.el8.nci.x86_64-x86_64-with-glibc2.28

Output dir: /g/data/nm31/d/r3.21_aatol_extra_samples_2022/spades_test Mode: read error correction and assembling Debug mode is turned OFF

Dataset parameters: Standard mode For multi-cell/isolate data we recommend to use '--isolate' option; for single-cell MDA data use '--sc'; for metagenomic data use '--meta'; for RNA-Seq use '--rna'. Reads: Traceback (most recent call last): File "/g/data/nm31/bin/SPAdes-3.15.2-Linux/bin/spades.py", line 651, in main(sys.argv) File "/g/data/nm31/bin/SPAdes-3.15.2-Linux/bin/spades.py", line 591, in main print_params(log, log_filename, command_line, args, cfg) File "/g/data/nm31/bin/SPAdes-3.15.2-Linux/bin/spades.py", line 327, in print_params print_used_values(cfg, log) File "/g/data/nm31/bin/SPAdes-3.15.2-Linux/bin/spades.py", line 117, in print_used_values dataset_data = pyyaml.load(open(cfg["dataset"].yaml_filename)) File "/g/data/nm31/bin/SPAdes-3.15.2-Linux/share/spades/pyyaml3/init.py", line 72, in load return loader.get_single_data() File "/g/data/nm31/bin/SPAdes-3.15.2-Linux/share/spades/pyyaml3/constructor.py", line 37, in get_single_data return self.construct_document(node) File "/g/data/nm31/bin/SPAdes-3.15.2-Linux/share/spades/pyyaml3/constructor.py", line 46, in construct_document for dummy in generator: File "/g/data/nm31/bin/SPAdes-3.15.2-Linux/share/spades/pyyaml3/constructor.py", line 398, in construct_yaml_map value = self.construct_mapping(node) File "/g/data/nm31/bin/SPAdes-3.15.2-Linux/share/spades/pyyaml3/constructor.py", line 204, in construct_mapping return super().construct_mapping(node, deep=deep) File "/g/data/nm31/bin/SPAdes-3.15.2-Linux/share/spades/pyyaml3/constructor.py", line 126, in construct_mapping if not isinstance(key, collections.Hashable): AttributeError: module 'collections' has no attribute 'Hashable'

[tallnuttrbgv]

Updated to spades 3.15.4, which works with python 3.10, and issue is now solved. Thanks.