Flye
Flye copied to clipboard
fenderglass
no consensus with flye-minimap2
Hello, I am having problems with the consensus step of Flye 2.9-b1774 installed in anaconda3, where the pipeline aborts with:
[2022-04-29 09:50:49] INFO: Starting Flye 2.9-b1774 [2022-04-29 09:50:50] INFO: >>>STAGE: configure [2022-04-29 09:50:50] INFO: Configuring run [2022-04-29 09:51:14] INFO: Total read length: 5487080319 [2022-04-29 09:51:14] INFO: Reads N50/N90: 7485 / 1239 [2022-04-29 09:51:14] INFO: Minimum overlap set to 1000 [2022-04-29 09:51:14] INFO: >>>STAGE: assembly [2022-04-29 09:51:14] INFO: Assembling disjointigs [2022-04-29 09:51:14] INFO: Reading sequences [2022-04-29 09:51:45] INFO: Building minimizer index [2022-04-29 09:51:45] INFO: Pre-calculating index storage 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [2022-04-29 09:53:30] INFO: Filling index 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [2022-04-29 09:55:20] INFO: Extending reads [2022-04-29 09:55:36] INFO: Overlap-based coverage: 3 [2022-04-29 09:55:36] INFO: Median overlap divergence: 0.086225 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [2022-04-29 10:13:02] INFO: Assembled 56488 disjointigs [2022-04-29 10:13:10] INFO: Generating sequence 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [2022-04-29 10:14:40] INFO: Filtering contained disjointigs 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% [2022-04-29 10:17:20] INFO: Contained seqs: 2765 [2022-04-29 10:17:45] INFO: >>>STAGE: consensus [2022-04-29 10:17:45] INFO: Running Minimap2 [2022-04-29 10:30:59] ERROR: Error running minimap2, terminating. See the alignment error log for details: /mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/minimap.stderr [2022-04-29 10:30:59] ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/00-assembly/draft_assembly.fasta' '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_raw_bonito.fa' -x map-ont -t 20 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/sort_220429_101745' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/minimap.bam'"]' returned non-zero exit status 137. [2022-04-29 10:30:59] ERROR: Pipeline aborted
and the error log minimap_stderr.log
When I run the command alone, without ' ' s, it completed, but then I was not able to resume the run using --resume-from consensus. Thank you for the help.
Hi,
Looks like minimap_stderr log is from a different run (different file names). Could you submit the matching one?
Sorry about that. I rerun flye after deleting the output folder and files. I renamed the .stderr file to be able to upload it to the thread. Here is the output from the consensus stage.
[2022-05-03 09:15:48] INFO: >>>STAGE: consensus [2022-05-03 09:15:48] INFO: Running Minimap2 [2022-05-03 09:36:37] ERROR: Error running minimap2, terminating. See the alignment error log for details: /mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/minimap.stderr [2022-05-03 09:36:37] ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/00-assembly/draft_assembly.fasta' '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_raw_bonito.fa' -x map-ont -t 20 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/sort_220503_091548' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cerasi/assemblies/assembly-flye2.9-CZ_17-T2_19/CZ_17-T2_19_nano/10-consensus/minimap.bam'"]' returned non-zero exit status 137.
Cheers
Hi fenderglass, in case it helps, I also ran an assembly on the same system as twolfe, but from a different user, and with different data, but generating the same error:
The versions: Flye = 2.9-b1768 Minimap2 = 2.24-r1122, both from conda.
The command:
flye --nano-hq 1_Cmel4_1_2_PCR1-2-3_SUP.fastq \
--out-dir /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/ \
--genome-size 438m \
--asm-coverage 34 \
--threads 24 \
--iterations 1
The error message:
[2022-05-03 10:51:02] ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 24 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/10-consensus/sort_220503_103626' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/10-consensus/minimap.bam'"]' returned non-zero exit status 137.
[2022-05-03 10:51:02] ERROR: Pipeline aborted
The minimap_stderror.log:
[samfaipath] build FASTA index...
[M::mm_idx_gen::21.332*1.48] collected minimizers
[M::mm_idx_gen::22.612*2.35] sorted minimizers
[M::main::22.622*2.34] loaded/built the index for 129893 target sequence(s)
[M::mm_mapopt_update::23.693*2.28] mid_occ = 1171
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 129893
[M::mm_idx_stat::24.373*2.25] distinct minimizers: 62359831 (42.64% are singletons); average occurrences: 5.557; average spacing: 5.241; total length: 1816104594
/bin/bash: line 1: 3473754 Killed flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 24 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G
3473755 Done | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/00-assembly/draft_assembly.fasta' -u -
3473756 Done | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/10-consensus/sort_220503_103626' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial3/10-consensus/minimap.bam'
Prior to the error message the readout is basically the same as in that of twolfe, and I also note that I can run flye with no problems to assemble smaller genomes in meta mode. I just add this in case it helps rule out possible options / to help debug ..
Very likely you are running out of memory. Can you share more details about the system and send flye.log? If it is a memory error, my only suggestion would be to try running with a reduced number of threads.
Here is the fly.log. I cut out the middle part and keep the first and last parts. flye.log Thanks for having a look.
Hi, yes, I also wondered about memory, have been working my down to fewer threads and reads, maybe still being to greedy, will try with fewer. As twolfe send a flye.log I won't also send one, but system has 125 Gb RAM, 48 cores, and a lot of hard disk space arranged in a RAID 10.
I see, I think it's a memory issue. The log says there are 128Gb in total, but only 56Gb was available at the time of run.
I'd try to run with more available RAM, and at the same time reduce the number of threads. Unfortunately I don't have much control over how much RAM minimap2 consumes..
Ah, interesting, yes, we will report back after trying with more RAM and fewer threads etc ;)
Alright, I've started a run in 12 threads, with 123 RAM available and using only 24x longest reads, will report on how it or even fewer core runs go ;)
so far:
nome-size 438m --asm-coverage 24 --threads 12 --iterations 1
[2022-05-03 16:53:36] root: DEBUG: Python version: 3.8.5 (default, Sep 4 2020,
07:30:14)
[GCC 7.3.0]
[2022-05-03 16:53:36] root: INFO: >>>STAGE: configure
[2022-05-03 16:53:36] root: INFO: Configuring run
[2022-05-03 16:55:46] root: INFO: Total read length: 23150251229
[2022-05-03 16:55:46] root: INFO: Input genome size: 438000000
[2022-05-03 16:55:46] root: INFO: Estimated coverage: 52
[2022-05-03 16:55:46] root: INFO: Reads N50/N90: 1746 / 566
[2022-05-03 16:55:46] root: INFO: Minimum overlap set to 1000
[2022-05-03 16:55:46] root: INFO: Using longest 24x reads for contig assembly
[2022-05-03 16:55:49] root: DEBUG: Min read length cutoff: 1962
[2022-05-03 16:55:49] root: INFO: >>>STAGE: assembly
[2022-05-03 16:55:49] root: INFO: Assembling disjointigs
[2022-05-03 16:55:49] root: DEBUG: -----Begin assembly log------
[2022-05-03 16:55:49] root: DEBUG: Running: flye-modules assemble --reads /mnt/r
aid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq --out-asm /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial4/00-assembly/draft_assembly.fasta --config /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_nano_hq.cfg --log /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial4/flye.log --threads 12 --genome-size 438000000 --min-ovlp 1000 --min-read 1962
[2022-05-03 16:55:49] DEBUG: Build date: Feb 22 2022 03:49:40
[2022-05-03 16:55:49] DEBUG: Total RAM: 125 Gb
[2022-05-03 16:55:49] DEBUG: Available RAM: 123 Gb
[2022-05-03 16:55:49] DEBUG: Total CPUs: 48
Hi again, So with more RAM (see flye.log), the consensus step moves a little further but still aborts (see minimap.stderr.log) flye.log minimap.stderr.log Notice I have a low coverage of around 3X for this particular R10 flow cell sequenced sample (--nano-hq parameter). Could this be the root of the problem? I have other samples with slightly higher coverage that I could combine with this sample, but they were sequenced on R9 flow cells. Cheers.
Hi fenderglass, another bit of extra information, in case of use. I've tried a couple more times, the most recent I set the command as before, but with 4 threads. Interestingly the log shows 123 GB available RAM, and notes that peak RAM usage was only 67 GB. Nonetheless, it run died with the general error as before, although getting a bit further in minimal:
log:
[2022-05-04 01:28:20] root: INFO: >>>STAGE: configure
[2022-05-04 01:28:20] root: INFO: Configuring run
[2022-05-04 01:30:28] root: INFO: Total read length: 23150251229
[2022-05-04 01:30:28] root: INFO: Input genome size: 438000000
[2022-05-04 01:30:28] root: INFO: Estimated coverage: 52
[2022-05-04 01:30:28] root: INFO: Reads N50/N90: 1746 / 566
[2022-05-04 01:30:28] root: INFO: Minimum overlap set to 1000
[2022-05-04 01:30:28] root: INFO: Using longest 40x reads for contig assembly
[2022-05-04 01:30:32] root: DEBUG: Min read length cutoff: 915
[2022-05-04 01:30:32] root: INFO: >>>STAGE: assembly
[2022-05-04 01:30:32] root: INFO: Assembling disjointigs
[2022-05-04 01:30:32] root: DEBUG: -----Begin assembly log------
[2022-05-04 01:30:32] root: DEBUG: Running: flye-modules assemble --reads /mnt/raid10/c
acopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel
4_1_2_PCR1-2-3_SUP.fastq --out-asm /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenome
s/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/00-assembly/draft_assembly.f
asta --config /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/a
sm_nano_hq.cfg --log /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_
PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/flye.log --threads 4 --genome-size 4380000
00 --min-ovlp 1000 --min-read 915
[2022-05-04 01:30:32] DEBUG: Build date: Feb 22 2022 03:49:40
[2022-05-04 01:30:32] DEBUG: Total RAM: 125 Gb
[2022-05-04 01:30:32] DEBUG: Available RAM: 123 Gb
[2022-05-04 01:30:32] DEBUG: Total CPUs: 48
...
[2022-05-04 06:00:20] INFO: Contained seqs: 10663
[2022-05-04 06:00:22] DEBUG: Writing FASTA
[2022-05-04 06:00:32] DEBUG: Peak RAM usage: 67 Gb
-----------End assembly log------------
[2022-05-04 06:00:50] root: DEBUG: Disjointigs length: 1832130745, N50: 22326
[2022-05-04 06:00:51] root: INFO: >>>STAGE: consensus
[2022-05-04 06:00:51] root: INFO: Running Minimap2
[2022-05-04 07:22:59] root: ERROR: Error running minimap2, terminating. See the alignment error log for details: /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/10-consensus/minimap.stderr
[2022-05-04 07:23:00] root: ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 4 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/10-consensus/sort_220504_060051' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/10-consensus/minimap.bam'"]' returned non-zero exit status 137.
[2022-05-04 07:23:00] root: ERROR: Pipeline aborted
minimap:
[samfaipath] build FASTA index...
[M::mm_idx_gen::21.216*1.47] collected minimizers
[M::mm_idx_gen::25.825*1.90] sorted minimizers
[M::main::25.843*1.90] loaded/built the index for 130701 target sequence(s)
[M::mm_mapopt_update::26.860*1.86] mid_occ = 1180
[M::mm_idx_stat] kmer size: 15; skip: 10; is_hpc: 0; #seq: 130701
[M::mm_idx_stat::27.512*1.84] distinct minimizers: 62534741 (42.47% are singletons); average occurrences: 5.590; average spacing: 5.241; total length: 1832130745
/bin/bash: line 1: 3500177 Killed flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 4 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G
3500178 Done | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/00-assembly/draft_assembly.fasta' -u -
3500179 Done | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/10-consensus/sort_220504_060051' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial5/10-consensus/minimap.bam'
That made me think this may not be a memory issue, I wondered if you would agree, or am I missing something there? Do you perhaps have any other ideas what could be tried?
Thanks for any suggestions :)
@JamesHowie14 the minimap2 error long say that the process was killed by the system. This is typically happens when you are running out of resources for a process. This is why I suspecting memory issues. But it could be also some limitation of the system itself (e.g. memory allocation per user etc).
- Could you give more information how about the system and how did you install Flye (e.g. bioconda package?)
- On linux you can check the system log (
dmesg) for out-of-memory records after the crash - I see that exit code 137 is commonly associated with Docker environments - are you using Docker? It can have a substantial memory overhead.
- Try running minimap2 command "manually" and watch the memory usage. This might also give you a more clear error message:
flye-minimap2 'path_to_draft_assembly.fasta' 'path_to_reads.fastq' -x map-ont -t 4 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T 'path_to_draft_assembly.fasta' -u - | flye-samtools sort -O bam -@ 4 -l 1 -m 4G -o 'minimap.bam
Dear Fenderglass,
Thanks for your response, OK so:
1 - the system is Linux x86_64, Ubuntu 20.04.1 LTS, with two sockets of 24 cpu, for a total 48 cpu, with 2 threads per core; the cpu max-min MHz are 4568-2200. In terms of RAM we have 126 GB. For hard disk storage, we have a 2TB solid state drive, and a RAID10 array of hard disks which gives 38-40 TB space. I have run all commands reading and writing directly to the RAID10, as for some assemblers hard drive space was an issue.
Flye version = 2.9-b1768, installed via conda
2 - to make sure my response is current I re-ran the command:
flye --nano-hq 1_Cmel4_1_2_PCR1-2-3_SUP.fastq \
--out-dir /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/ \
--genome-size 438m \
--asm-coverage 10 \
--threads 12 \
--iterations 1
, as expected, this recovered the error:
[2022-05-16 14:16:12] INFO: Filtering contained disjointigs
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
[2022-05-16 14:22:56] INFO: Contained seqs: 1699
[2022-05-16 14:23:22] INFO: >>>STAGE: consensus
[2022-05-16 14:23:22] INFO: Running Minimap2
[2022-05-16 14:36:38] ERROR: Error running minimap2, terminating. See the alignment error log for details: /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/10-consensus/minimap.stderr
[2022-05-16 14:36:39] ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 12 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/10-consensus/sort_220516_142322' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/10-consensus/minimap.bam'"]' returned non-zero exit status 137.
[2022-05-16 14:36:39] ERROR: Pipeline aborted
Given that, I checked "dmesg", which shows:
[9935276.338517] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1007.slice/session-21465.scope,task=flye-minimap2,pid=618696,uid=1007
[9935276.338536] Out of memory: Killed process 618696 (flye-minimap2) total-vm:46467540kB, anon-rss:44820240kB, file-rss:0kB, shmem-rss:0kB, UID:1007 pgtables:89228kB oom_score_adj:0
[9935278.690076] oom_reaper: reaped process 618696 (flye-minimap2), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
The full message was:
[9935276.337889] systemd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[9935276.337897] CPU: 30 PID: 1 Comm: systemd Tainted: P OE 5.13.0-27-generic #29~20.04.1-Ubuntu
[9935276.337900] Hardware name: Micro-Star International Co., Ltd. MS-7C60/TRX40 PRO 10G (MS-7C60), BIOS 1.60 05/13/2020
[9935276.337901] Call Trace:
[9935276.337906] dump_stack+0x7d/0x9c
[9935276.337911] dump_header+0x4f/0x1f6
[9935276.337915] oom_kill_process.cold+0xb/0x10
[9935276.337918] out_of_memory.part.0+0x1fa/0x480
[9935276.337922] out_of_memory+0x6d/0xd0
[9935276.337924] __alloc_pages_slowpath.constprop.0+0xc86/0xd50
[9935276.337928] __alloc_pages+0x2ce/0x320
[9935276.337931] alloc_pages+0x87/0xe0
[9935276.337933] __page_cache_alloc+0x89/0xb0
[9935276.337937] pagecache_get_page+0x218/0x4d0
[9935276.337940] filemap_fault+0x5bf/0x9b0
[9935276.337943] ? do_set_pte+0xc8/0x140
[9935276.337947] ? next_uptodate_page+0x160/0x270
[9935276.337949] ext4_filemap_fault+0x32/0x50
[9935276.337953] __do_fault+0x3e/0xc0
[9935276.337956] do_fault+0x1ea/0x440
[9935276.337958] __handle_mm_fault+0x619/0x8e0
[9935276.337961] handle_mm_fault+0xda/0x2b0
[9935276.337963] do_user_addr_fault+0x1bb/0x650
[9935276.337966] exc_page_fault+0x7d/0x170
[9935276.337969] ? asm_exc_page_fault+0x8/0x30
[9935276.337972] asm_exc_page_fault+0x1e/0x30
[9935276.337974] RIP: 0033:0x7f6a5de0a4f8
[9935276.337980] Code: Unable to access opcode bytes at RIP 0x7f6a5de0a4ce.
[9935276.337981] RSP: 002b:00007fff5a127dc0 EFLAGS: 00010207
[9935276.337983] RAX: 00007f6a5d1d6900 RBX: 00007fff5a127e07 RCX: 0000000000000001
[9935276.337985] RDX: 0000000000000065 RSI: 00007fff5a127e07 RDI: 000055734b93ee30
[9935276.337986] RBP: 000055734b93ee30 R08: 0000000000000001 R09: 0000000000000001
[9935276.337987] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
[9935276.337988] R13: 0000000000100000 R14: 0000000000000006 R15: 00007fff5a127e07
[9935276.337990] Mem-Info:
[9935276.337994] active_anon:2557632 inactive_anon:29780650 isolated_anon:0
active_file:2292 inactive_file:2347 isolated_file:158
unevictable:0 dirty:1 writeback:0
slab_reclaimable:125241 slab_unreclaimable:146230
mapped:656 shmem:1064 pagetables:70424 bounce:0
free:145429 free_pcp:63 free_cma:0
[9935276.337998] Node 0 active_anon:10230528kB inactive_anon:119122600kB active_file:9168kB inactive_file:9388kB unevictable:0kB isolated(anon):0kB isolated(file):632kB mapped:2624kB dirty:4kB writeback:0kB shmem:4256kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:28304kB pagetables:281696kB all_unreclaimable? no
[9935276.338003] Node 0 DMA free:11280kB min:4kB low:16kB high:28kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15376kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[9935276.338008] lowmem_reserve[]: 0 2842 128608 128608 128608
[9935276.338012] Node 0 DMA32 free:504404kB min:1492kB low:4400kB high:7308kB reserved_highatomic:0KB active_anon:313668kB inactive_anon:2111728kB active_file:0kB inactive_file:28kB unevictable:0kB writepending:0kB present:3035588kB managed:2969524kB mlocked:0kB bounce:0kB free_pcp:252kB local_pcp:0kB free_cma:0kB
[9935276.338016] lowmem_reserve[]: 0 0 125765 125765 125765
[9935276.338019] Node 0 Normal free:66032kB min:66080kB low:194864kB high:323648kB reserved_highatomic:0KB active_anon:9916860kB inactive_anon:117010872kB active_file:9172kB inactive_file:8944kB unevictable:0kB writepending:4kB present:131057664kB managed:128792996kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[9935276.338024] lowmem_reserve[]: 0 0 0 0 0
[9935276.338026] Node 0 DMA: 2*4kB (U) 1*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11280kB
[9935276.338036] Node 0 DMA32: 850*4kB (UE) 716*8kB (UME) 1476*16kB (UME) 1365*32kB (UE) 1079*64kB (UME) 704*128kB (UME) 341*256kB (UME) 145*512kB (UME) 51*1024kB (UE) 9*2048kB (UME) 9*4096kB (UME) = 504648kB
[9935276.338050] Node 0 Normal: 3603*4kB (UME) 1701*8kB (UME) 497*16kB (UME) 575*32kB (UME) 92*64kB (UME) 9*128kB (UM) 0*256kB 3*512kB (M) 0*1024kB 2*2048kB (ME) 0*4096kB = 67044kB
[9935276.338064] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[9935276.338066] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[9935276.338068] 21108 total pagecache pages
[9935276.338068] 14730 pages in swap cache
[9935276.338069] Swap cache stats: add 126197531, delete 126187441, find 33131458/43601237
[9935276.338071] Free swap = 0kB
[9935276.338071] Total swap = 2097148kB
[9935276.338072] 33527312 pages RAM
[9935276.338073] 0 pages HighMem/MovableOnly
[9935276.338074] 582838 pages reserved
[9935276.338074] 0 pages hwpoisoned
[9935276.338075] Tasks state (memory values in pages):
[9935276.338076] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[9935276.338089] [ 785] 0 785 25415 254 180224 461 -250 systemd-journal
[9935276.338093] [ 811] 0 811 6173 658 73728 367 -1000 systemd-udevd
[9935276.338097] [ 1411] 102 1411 22564 27 86016 184 0 systemd-timesyn
[9935276.338100] [ 1445] 0 1445 62331 804 102400 87 0 accounts-daemon
[9935276.338102] [ 1446] 0 1446 637 0 45056 47 0 acpid
[9935276.338105] [ 1449] 115 1449 2170 112 53248 67 0 avahi-daemon
[9935276.338107] [ 1453] 0 1453 4513 26 61440 40 0 cron
[9935276.338109] [ 1456] 103 1456 2562 617 61440 110 -900 dbus-daemon
[9935276.338112] [ 1457] 0 1457 155830 529 184320 382 0 NetworkManager
[9935276.338114] [ 1464] 0 1464 20533 109 65536 51 0 irqbalance
[9935276.338116] [ 1481] 0 1481 61060 3 94208 159 0 switcheroo-cont
[9935276.338119] [ 1484] 0 1484 4425 329 77824 173 0 systemd-logind
[9935276.338121] [ 1490] 0 1490 98519 907 135168 223 0 udisksd
[9935276.338123] [ 1495] 0 1495 3420 24 65536 135 0 wpa_supplicant
[9935276.338126] [ 1513] 115 1513 2086 1 53248 81 0 avahi-daemon
[9935276.338128] [ 1550] 121 1550 82281 364 122880 1078 0 colord
[9935276.338130] [ 1597] 0 1597 62156 0 110592 538 0 gdm3
[9935276.338132] [ 1783] 0 1783 31690 0 143360 2070 0 unattended-upgr
[9935276.338135] [ 1784] 0 1784 43827 121 110592 203 0 gdm-session-wor
[9935276.338137] [ 1828] 125 1828 4749 272 69632 255 0 systemd
[9935276.338139] [ 1830] 125 1830 42317 1 94208 908 0 (sd-pam)
[9935276.338141] [ 1887] 125 1887 241141 46 196608 1061 0 pulseaudio
[9935276.338144] [ 1889] 125 1889 183309 244 221184 1962 0 tracker-miner-f
[9935276.338146] [ 1894] 125 1894 43161 1 86016 172 0 gdm-x-session
[9935276.338148] [ 1898] 125 1898 652280 1065 643072 7983 0 Xorg
[9935276.338150] [ 1906] 125 1906 1866 109 57344 76 0 dbus-daemon
[9935276.338152] [ 1909] 111 1909 38233 38 61440 37 0 rtkit-daemon
[9935276.338154] [ 1928] 125 1928 62087 109 94208 145 0 gvfsd
[9935276.338156] [ 1949] 125 1949 78202 0 94208 223 0 gvfsd-fuse
[9935276.338158] [ 1959] 125 1959 80701 493 106496 158 0 gvfs-udisks2-vo
[9935276.338161] [ 2474] 125 2474 1323 0 49152 48 0 dbus-run-sessio
[9935276.338163] [ 2482] 125 2482 1891 0 57344 246 0 dbus-daemon
[9935276.338165] [ 2486] 125 2486 123117 152 167936 422 0 gnome-session-b
[9935276.338167] [ 2689] 125 2689 76378 0 94208 201 0 at-spi-bus-laun
[9935276.338169] [ 2699] 125 2699 1810 0 53248 119 0 dbus-daemon
[9935276.338171] [ 2877] 125 2877 1571953 10369 1486848 15048 0 gnome-shell
[9935276.338174] [ 2946] 125 2946 61127 28 90112 169 0 gvfs-goa-volume
[9935276.338176] [ 2950] 125 2950 138950 142 258048 1408 0 goa-daemon
[9935276.338178] [ 2969] 125 2969 80847 56 114688 385 0 goa-identity-se
[9935276.338180] [ 2974] 125 2974 61719 28 90112 241 0 gvfs-gphoto2-vo
[9935276.338182] [ 2979] 125 2979 61116 52 102400 159 0 gvfs-mtp-volume
[9935276.338184] [ 2989] 125 2989 81339 74 110592 214 0 gvfs-afc-volume
[9935276.338186] [ 2995] 0 2995 81650 196 110592 255 0 upowerd
[9935276.338188] [ 3028] 125 3028 79891 0 106496 379 0 ibus-daemon
[9935276.338191] [ 3031] 125 3031 42834 0 86016 168 0 ibus-memconf
[9935276.338193] [ 3034] 125 3034 49147 0 143360 521 0 ibus-x11
[9935276.338195] [ 3036] 125 3036 61284 1 94208 180 0 ibus-portal
[9935276.338197] [ 3046] 125 3046 40726 0 81920 230 0 at-spi2-registr
[9935276.338199] [ 3050] 125 3050 61057 0 94208 155 0 xdg-permission-
[9935276.338201] [ 3063] 125 3063 734706 0 282624 1538 0 gjs
[9935276.338203] [ 3083] 125 3083 118318 164 126976 213 0 gsd-sharing
[9935276.338205] [ 3085] 125 3085 85920 0 155648 599 0 gsd-wacom
[9935276.338207] [ 3086] 125 3086 105170 253 155648 771 0 gsd-color
[9935276.338209] [ 3087] 125 3087 86044 130 159744 417 0 gsd-keyboard
[9935276.338211] [ 3088] 125 3088 64378 102 131072 447 0 gsd-print-notif
[9935276.338214] [ 3090] 125 3090 116433 120 118784 107 0 gsd-rfkill
[9935276.338216] [ 3093] 125 3093 80948 0 114688 292 0 gsd-smartcard
[9935276.338218] [ 3094] 125 3094 95559 0 172032 554 0 gsd-datetime
[9935276.338220] [ 3096] 125 3096 171747 147 188416 566 0 gsd-media-keys
[9935276.338222] [ 3101] 125 3101 61035 0 98304 182 0 gsd-screensaver
[9935276.338224] [ 3104] 125 3104 81993 0 118784 310 0 gsd-sound
[9935276.338226] [ 3105] 125 3105 79572 0 106496 202 0 gsd-a11y-settin
[9935276.338228] [ 3109] 125 3109 80087 108 114688 146 0 gsd-housekeepin
[9935276.338230] [ 3110] 125 3110 42832 0 77824 197 0 ibus-engine-sim
[9935276.338233] [ 3113] 125 3113 104705 159 159744 465 0 gsd-power
[9935276.338235] [ 3152] 125 3152 87704 0 159744 544 0 gsd-printer
[9935276.338237] [ 3402] 125 3402 39032 0 69632 170 0 dconf-service
[9935276.338239] [ 3409] 0 3409 1989907 3474 1241088 8017 -500 dockerd
[9935276.338241] [ 3419] 120 3419 81712 247 135168 315 0 whoopsie
[9935276.338243] [ 3424] 116 3424 2814 25 65536 94 0 kerneloops
[9935276.338245] [ 3426] 116 3426 2814 25 61440 94 0 kerneloops
[9935276.338248] [ 3430] 0 3430 1078080 1084 581632 635 0 teamviewerd
[9935276.338250] [ 3435] 0 3435 45749 529 135168 567 0 rserver
[9935276.338252] [ 3625] 0 3625 624 0 45056 21 0 bpfilter_umh
[9935276.338254] [ 3842] 1006 3842 4745 222 81920 305 0 systemd
[9935276.338256] [ 3847] 1006 3847 42321 1 94208 930 0 (sd-pam)
[9935276.338258] [ 3890] 1006 3890 265693 888 241664 2062 0 tracker-miner-f
[9935276.338260] [ 3900] 1006 3900 1896 96 53248 102 0 dbus-daemon
[9935276.338262] [ 3929] 1006 3929 62087 0 94208 282 0 gvfsd
[9935276.338264] [ 3934] 1006 3934 94586 0 98304 244 0 gvfsd-fuse
[9935276.338266] [ 3940] 1006 3940 80701 481 102400 171 0 gvfs-udisks2-vo
[9935276.338268] [ 3960] 1006 3960 61127 0 90112 231 0 gvfs-goa-volume
[9935276.338270] [ 3967] 1006 3967 138950 142 266240 1406 0 goa-daemon
[9935276.338272] [ 3997] 1006 3997 80847 67 114688 374 0 goa-identity-se
[9935276.338274] [ 4019] 1006 4019 61719 0 98304 273 0 gvfs-gphoto2-vo
[9935276.338276] [ 4024] 1006 4024 61116 80 98304 150 0 gvfs-mtp-volume
[9935276.338278] [ 4029] 1006 4029 81339 78 114688 226 0 gvfs-afc-volume
[9935276.338280] [ 4156] 0 4156 827 48 45056 25 0 mdadm
[9935276.338282] [ 4296] 0 4296 327885 53 413696 655 0 mk_manager_svc
[9935276.338285] [ 4300] 0 4300 39006 0 69632 84 0 crashpad_handle
[9935276.338287] [ 4320] 0 4320 408872 188 544768 6731 0 guppy_basecall_
[9935276.338289] [ 4321] 0 4321 114800 3 143360 530 0 grpcwebproxy
[9935276.338291] [ 4322] 0 4322 171301 0 282624 569 0 basecall_manage
[9935276.338293] [ 4330] 0 4330 39006 0 77824 74 0 crashpad_handle
[9935276.338295] [ 4343] 0 4343 114384 0 135168 519 0 grpcwebproxy
[9935276.338297] [ 4421] 1000 4421 4729 276 77824 252 0 systemd
[9935276.338299] [ 4422] 1000 4422 42353 1 94208 968 0 (sd-pam)
[9935276.338301] [ 4430] 1000 4430 183774 663 217088 2659 0 tracker-miner-f
[9935276.338304] [ 4439] 1000 4439 1900 97 53248 86 0 dbus-daemon
[9935276.338306] [ 4469] 1000 4469 62086 107 98304 153 0 gvfsd
[9935276.338308] [ 4474] 1000 4474 94586 0 102400 227 0 gvfsd-fuse
[9935276.338310] [ 4481] 1000 4481 80703 497 118784 165 0 gvfs-udisks2-vo
[9935276.338312] [ 4505] 1000 4505 61127 88 90112 105 0 gvfs-goa-volume
[9935276.338314] [ 4510] 1000 4510 138952 144 249856 1404 0 goa-daemon
[9935276.338316] [ 4533] 1000 4533 80847 61 110592 410 0 goa-identity-se
[9935276.338318] [ 4539] 1000 4539 61718 65 106496 201 0 gvfs-gphoto2-vo
[9935276.338320] [ 4544] 1000 4544 61116 83 98304 130 0 gvfs-mtp-volume
[9935276.338322] [ 4553] 1000 4553 81339 74 118784 213 0 gvfs-afc-volume
[9935276.338324] [ 4714] 1000 4714 5158 23 53248 652 0 tmux: server
[9935276.338326] [ 4715] 1000 4715 5015 1 69632 546 0 bash
[9935276.338329] [ 17574] 1006 17574 117308 3251 352256 17318 0 tracker-store
[9935276.338331] [ 17644] 1000 17644 42723 51 86016 129 0 gvfsd-metadata
[9935276.338333] [ 17653] 125 17653 42721 79 86016 100 0 gvfsd-metadata
[9935276.338335] [ 17915] 1006 17915 42722 79 86016 119 0 gvfsd-metadata
[9935276.338337] [ 24034] 1006 24034 7621 2932 81920 204 0 tmux: server
[9935276.338340] [ 24035] 1006 24035 5841 202 69632 876 0 bash
[9935276.338342] [ 40965] 1006 40965 5296 187 69632 668 0 bash
[9935276.338344] [ 511345] 0 511345 3044 47 61440 186 -1000 sshd
[9935276.338347] [ 699861] 1006 699861 5146 414 69632 263 0 bash
[9935276.338350] [3777985] 1006 3777985 4950 278 61440 225 0 bash
[9935276.338352] [4169811] 1006 4169811 5191 2 65536 743 0 bash
[9935276.338355] [ 424928] 107 424928 4807 0 65536 51 0 uuidd
[9935276.338357] [ 600399] 1006 600399 4999 161 69632 383 0 bash
[9935276.338359] [ 731524] 0 731524 58799 785 90112 184 0 polkitd
[9935276.338362] [ 731528] 0 731528 78465 130 114688 351 0 ModemManager
[9935276.338364] [ 752141] 0 752141 1129976 2354 729088 4073 -999 containerd
[9935276.338366] [1029020] 1006 1029020 96989 0 118784 334 0 gnome-keyring-d
[9935276.338369] [1314442] 1006 1314442 2943 486 61440 73 0 bash
[9935276.338371] [2539261] 101 2539261 6038 146 94208 946 0 systemd-resolve
[9935276.338374] [2708297] 0 2708297 3497 3 69632 370 0 sshd
[9935276.338376] [2708421] 1006 2708421 3842 108 73728 694 0 sshd
[9935276.338378] [2708422] 1006 2708422 2835 2 61440 446 0 bash
[9935276.338380] [2708460] 1006 2708460 2520 12 53248 77 0 tmux: client
[9935276.338382] [3491607] 1007 3491607 4717 266 73728 248 0 systemd
[9935276.338384] [3491608] 1007 3491608 42668 97 98304 1206 0 (sd-pam)
[9935276.338387] [3491617] 1007 3491617 181696 680 212992 1955 0 tracker-miner-f
[9935276.338389] [3491642] 1007 3491642 1844 96 53248 58 0 dbus-daemon
[9935276.338391] [3491663] 1007 3491663 60012 119 102400 130 0 gvfsd
[9935276.338394] [3491674] 1007 3491674 94586 0 90112 194 0 gvfsd-fuse
[9935276.338396] [3491686] 1007 3491686 78629 463 114688 152 0 gvfs-udisks2-vo
[9935276.338398] [3491712] 1007 3491712 59053 0 86016 234 0 gvfs-goa-volume
[9935276.338400] [3491716] 1007 3491716 137415 137 262144 1464 0 goa-daemon
[9935276.338402] [3491725] 1007 3491725 78773 48 114688 275 0 goa-identity-se
[9935276.338405] [3491730] 1007 3491730 59644 0 94208 271 0 gvfs-gphoto2-vo
[9935276.338407] [3491735] 1007 3491735 59041 74 94208 131 0 gvfs-mtp-volume
[9935276.338409] [3491739] 1007 3491739 79265 59 110592 219 0 gvfs-afc-volume
[9935276.338411] [3492310] 1007 3492310 40646 67 86016 96 0 gvfsd-metadata
[9935276.338413] [3507273] 1007 3507273 8968 71 106496 5717 0 perl
[9935276.338415] [3558222] 0 3558222 10058 8 118784 2199 0 networkd-dispat
[9935276.338417] [3582405] 1006 3582405 3732302 2772533 26415104 398436 0 whatshap
[9935276.338420] [3640664] 104 3640664 56088 301 86016 0 0 rsyslogd
[9935276.338423] [ 197286] 1007 197286 2765 323 61440 20 0 tmux: server
[9935276.338425] [ 197287] 1007 197287 2876 293 61440 209 0 bash
[9935276.338427] [ 213280] 1007 213280 2875 199 65536 273 0 bash
[9935276.338430] [ 508741] 0 508741 7277 472 90112 19 0 cupsd
[9935276.338432] [ 508743] 0 508743 44761 573 114688 0 0 cups-browsed
[9935276.338434] [ 508756] 7 508756 3833 190 69632 9 0 dbus
[9935276.338437] [ 586286] 1007 586286 2876 1 57344 485 0 bash
[9935276.338440] [ 604958] 0 604958 3431 368 69632 0 0 sshd
[9935276.338442] [ 604991] 0 604991 698302 3447 466944 0 -900 snapd
[9935276.338444] [ 605033] 1007 605033 15313 10041 172032 1228 0 flye
[9935276.338447] [ 605656] 1006 605656 3506 371 69632 0 0 sshd
[9935276.338449] [ 605657] 1006 605657 2835 436 61440 0 0 bash
[9935276.338451] [ 606603] 0 606603 3503 366 65536 0 0 sshd
[9935276.338453] [ 606669] 1007 606669 3503 373 65536 0 0 sshd
[9935276.338455] [ 606670] 1007 606670 2864 468 57344 0 0 bash
[9935276.338457] [ 607423] 1007 607423 2525 71 53248 0 0 tmux: client
[9935276.338459] [ 608715] 0 608715 3504 367 65536 0 0 sshd
[9935276.338461] [ 608773] 1007 608773 3504 375 65536 0 0 sshd
[9935276.338464] [ 608774] 1007 608774 2864 468 53248 0 0 bash
[9935276.338466] [ 609926] 1007 609926 2525 71 53248 0 0 tmux: client
[9935276.338468] [ 614478] 1007 614478 1607528 9870 462848 0 0 circlator
[9935276.338471] [ 616253] 1007 616253 656 20 45056 0 0 sh
[9935276.338473] [ 616254] 1007 616254 656 30 45056 0 0 mhap.sh
[9935276.338475] [ 616286] 1007 616286 656 30 45056 0 0 mhap.sh
[9935276.338478] [ 616287] 1007 616287 122345 121115 1011712 0 0 mhapConvert
[9935276.338480] [ 616288] 1007 616288 16052969 8997646 73019392 0 0 java
[9935276.338482] [ 618695] 1007 618695 2461 61 53248 0 0 bash
[9935276.338485] [ 618696] 1007 618696 11616885 11205060 91369472 0 0 flye-minimap2
[9935276.338487] [ 618697] 1007 618697 1804 442 53248 0 0 flye-samtools
[9935276.338489] [ 618698] 1007 618698 1291 31 45056 0 0 flye-samtools
[9935276.338492] [ 618791] 1007 618791 656 19 40960 0 0 sh
[9935276.338494] [ 618792] 1007 618792 656 19 49152 0 0 spades.py
[9935276.338496] [ 618794] 1007 618794 5874 1940 81920 0 0 spades.py
[9935276.338498] [ 619278] 1007 619278 656 20 45056 0 0 sh
[9935276.338500] [ 619279] 1007 619279 656 28 45056 0 0 mhap.sh
[9935276.338502] [ 619311] 1007 619311 656 30 45056 0 0 mhap.sh
[9935276.338504] [ 619312] 1007 619312 122345 121116 1024000 0 0 mhapConvert
[9935276.338506] [ 619313] 1007 619313 16036328 8995063 73019392 0 0 java
[9935276.338509] [ 619897] 1007 619897 11162 6284 118784 0 0 spades-correcto
[9935276.338511] [ 619916] 1007 619916 656 20 40960 0 0 sh
[9935276.338513] [ 619917] 1007 619917 76500 23563 282624 0 0 bwa
[9935276.338515] [ 619955] 0 619955 4513 26 53248 40 0 cron
[9935276.338517] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1007.slice/session-21465.scope,task=flye-minimap2,pid=618696,uid=1007
[9935276.338536] Out of memory: Killed process 618696 (flye-minimap2) total-vm:46467540kB, anon-rss:44820240kB, file-rss:0kB, shmem-rss:0kB, UID:1007 pgtables:89228kB oom_score_adj:0
[9935278.690076] oom_reaper: reaped process 618696 (flye-minimap2), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
, which looks to me pretty relevant. The job is killed due to memory, as you suspected. Although, being unfamiliar with this error read out, I am not sure if this is an actual hard limit, or if the mems_allowed=0 somehow limits the job instead.
3 - interesting to know, but no, I am not using docker; just running the command directly.
4 - I have also tried to run the command directly, as you suggested, which yeilds the error:
flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 4 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial8/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -O bam -@ 4 -l 1 -m 4G -o 'minimap.bam'
I watched, as the job ran and and memory slowly increased from around 24 % while it is showing "[M::mm_idx_stat::26.837*1.99] distinct minimizers" to 25, 26, .. 30, .. 34%.
I looked away for a couple of minutes, then the job died.
Notably there were several other processes running which had 27.4 + 27.4 + 8.7 = 63.2 % of RAM used, meaning 36.8% remained free, so I guess we then reached that limit while I looked away.
In terms of clearer error messages from the direct command,
I have tried before, when the other processes were not present, which led to the same error, presumably after reaching a higher memory allocation rather than clipping out at 37% or so.
All together then, it looks like memory, as you suggested.
Given that, I wondered if you have a suggestion on roughly how much memory is needed for a 450 Mb genome that is highly repetitive. I had thought we would have enough given the general guidelines, but I can look to run on a cluster node with more RAM.
@JamesHowie14 normally minimap2 rarely takes more than 64Gb RAM even for a human-sized genome. I think the problem is that your genome just happened to be very repetitive, and this resulted in a lot of duplication in disjointigs, which the results in high memory usage by minimap2.
What kind of genome is it? In the flye.log file you sent previously, your genome size is set to 2.5G, is this accurate (since you mentioned it should be 450mb)?
Your read mean error rate is about 8%, which is closer to nano-raw mode. I would try running with --nano-raw, and maybe adding -m 5000, which should reduce duplications.
Hi fendeglass, thanks for your reply. OK, interesting. Yes, from another Illumina dataset we estimate the genome as about 50% repetitive. The estimated genome size is about 450mb, I think my Flye log it is set to 438 [the exact "expected" size]. Tom's one is 2.5G. Mine is a psyllid, sort of like a leaf hopper. I'll try again with nano-raw, as you suggest, and also with -m 5000, and get back to you ;) Cheers!
Dear Fenderglass, so I've now tried that. First, I ran:
flye --nano-raw 1_Cmel4_1_2_PCR1-2-3_SUP.fastq \
--out-dir /mnt/raid10/.../Cmel4_FlyeTrial13/ \
--genome-size 438m \
--asm-coverage 50 \
-m 5000 \
--threads 12 \
--iterations 1
It runs, but then at the minimap2 step the memory use just goes up, up, up, till it maxes out and crashes. I also tried a 2 thread version, with the same result, just after a slower memory build up.
The 12 thread readout was:
[2022-06-10 15:25:10] root: INFO: Starting Flye 2.9-b1768
[2022-06-10 15:25:10] root: DEBUG: Cmd: /home/jhowie/miniconda3/bin/flye --nano-raw 1_Cmel4_1_2_PCR1-2-3_SUP.fastq --out-dir /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/ --genome-size 438m --asm-coverage 50 -m 5000 --threads 12 --iterations 1
[2022-06-10 15:25:10] root: DEBUG: Python version: 3.8.5 (default, Sep 4 2020, 07:30:14)
[GCC 7.3.0]
[2022-06-10 15:25:10] root: INFO: >>>STAGE: configure
[2022-06-10 15:25:10] root: INFO: Configuring run
[2022-06-10 15:27:18] root: INFO: Total read length: 23150251229
[2022-06-10 15:27:18] root: INFO: Input genome size: 438000000
[2022-06-10 15:27:18] root: INFO: Estimated coverage: 52
[2022-06-10 15:27:18] root: INFO: Reads N50/N90: 1746 / 566
[2022-06-10 15:27:18] root: INFO: Selected minimum overlap: 5000
[2022-06-10 15:27:18] root: INFO: Using longest 50x reads for contig assembly
[2022-06-10 15:27:23] root: DEBUG: Min read length cutoff: 444
[2022-06-10 15:27:23] root: INFO: >>>STAGE: assembly
[2022-06-10 15:27:23] root: INFO: Assembling disjointigs
[2022-06-10 15:27:23] root: DEBUG: -----Begin assembly log------
[2022-06-10 15:27:23] root: DEBUG: Running: flye-modules assemble --reads /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq --out-asm /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/00-assembly/draft_assembly.fasta --config /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_raw_reads.cfg --log /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/flye.log --threads 12 --genome-size 438000000 --min-ovlp 5000 --min-read 444
[2022-06-10 15:27:23] DEBUG: Build date: Feb 22 2022 03:49:40
[2022-06-10 15:27:23] DEBUG: Total RAM: 125 Gb
[2022-06-10 15:27:23] DEBUG: Available RAM: 123 Gb
[2022-06-10 15:27:23] DEBUG: Total CPUs: 48
[2022-06-10 15:27:23] DEBUG: Loading /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_raw_reads.cfg
[2022-06-10 15:27:23] DEBUG: Loading /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_defaults.cfg
[2022-06-10 15:27:23] DEBUG: big_genome_threshold=29000000
[2022-06-10 15:27:23] DEBUG: meta_read_filter_kmer_freq=100
[2022-06-10 15:27:23] DEBUG: chain_large_gap_penalty=2
[2022-06-10 15:27:23] DEBUG: chain_small_gap_penalty=0.5
[2022-06-10 15:27:23] DEBUG: chain_gap_jump_threshold=100
[2022-06-10 15:27:23] DEBUG: max_coverage_drop_rate=5
[2022-06-10 15:27:23] DEBUG: max_extensions_drop_rate=5
[2022-06-10 15:27:23] DEBUG: chimera_window=100
[2022-06-10 15:27:23] DEBUG: chimera_overhang=1000
[2022-06-10 15:27:23] DEBUG: min_reads_in_disjointig=4
[2022-06-10 15:27:23] DEBUG: max_inner_reads=10
[2022-06-10 15:27:23] DEBUG: max_inner_fraction=0.25
[2022-06-10 15:27:23] DEBUG: max_separation=500
[2022-06-10 15:27:23] DEBUG: unique_edge_length=50000
[2022-06-10 15:27:23] DEBUG: min_repeat_res_support=0.51
[2022-06-10 15:27:23] DEBUG: out_paths_ratio=5
[2022-06-10 15:27:23] DEBUG: graph_cov_drop_rate=5
[2022-06-10 15:27:23] DEBUG: coverage_estimate_window=100
[2022-06-10 15:27:23] DEBUG: max_bubble_length=50000
[2022-06-10 15:27:23] DEBUG: loop_coverage_rate=1.5
[2022-06-10 15:27:23] DEBUG: repeat_edge_cov_mult=1.75
[2022-06-10 15:27:23] DEBUG: weak_detach_rate=5
[2022-06-10 15:27:23] DEBUG: tip_coverage_rate=2
[2022-06-10 15:27:23] DEBUG: tip_length_rate=2
[2022-06-10 15:27:23] DEBUG: output_gfa_before_rr=0
[2022-06-10 15:27:23] DEBUG: low_cutoff_warning=1
[2022-06-10 15:27:23] DEBUG: kmer_size=17
[2022-06-10 15:27:23] DEBUG: use_minimizers=0
[2022-06-10 15:27:23] DEBUG: reads_base_alignment=0
[2022-06-10 15:27:23] DEBUG: meta_read_top_kmer_rate=0.40
[2022-06-10 15:27:23] DEBUG: maximum_jump=1500
[2022-06-10 15:27:23] DEBUG: maximum_overhang=1500
[2022-06-10 15:27:23] DEBUG: repeat_kmer_rate=100
[2022-06-10 15:27:23] DEBUG: assemble_ovlp_divergence=0.10
[2022-06-10 15:27:23] DEBUG: assemble_divergence_relative=1
[2022-06-10 15:27:23] DEBUG: repeat_graph_ovlp_divergence=0.08
[2022-06-10 15:27:23] DEBUG: read_align_ovlp_divergence=0.25
[2022-06-10 15:27:23] DEBUG: hpc_scoring_on=0
[2022-06-10 15:27:23] DEBUG: add_unassembled_reads=0
[2022-06-10 15:27:23] DEBUG: extend_contigs_with_repeats=0
[2022-06-10 15:27:23] DEBUG: min_read_cov_cutoff=3
[2022-06-10 15:27:23] DEBUG: short_tip_length=20000
[2022-06-10 15:27:23] DEBUG: long_tip_length=100000
[2022-06-10 15:27:23] DEBUG: Running with k-mer size: 17
[2022-06-10 15:27:23] DEBUG: Running with minimum overlap 5000
[2022-06-10 15:27:23] DEBUG: Metagenome mode: N
[2022-06-10 15:27:23] DEBUG: Short mode: N
[2022-06-10 15:27:23] INFO: Reading sequences
[2022-06-10 15:28:28] DEBUG: Building positional index
[2022-06-10 15:28:31] DEBUG: Total sequence: 3915919669 bp
[2022-06-10 15:28:32] INFO: Counting k-mers:
[2022-06-10 15:29:47] DEBUG: Updating k-mer histogram
[2022-06-10 15:30:52] DEBUG: Hash size: 23322863
[2022-06-10 15:30:52] DEBUG: Total k-mers 904133692
[2022-06-10 15:30:53] INFO: Filling index table (1/2)
[2022-06-10 15:33:02] DEBUG: Mean k-mer frequency: 30.1135
[2022-06-10 15:33:02] DEBUG: Repetitive k-mer frequency: 3011
[2022-06-10 15:33:02] DEBUG: Filtered 382356617 repetitive k-mers (0.24751)
[2022-06-10 15:33:04] INFO: Filling index table (2/2)
[2022-06-10 15:35:19] DEBUG: Sorting k-mer index
[2022-06-10 15:35:29] DEBUG: Selected k-mers: 69477999
[2022-06-10 15:35:29] DEBUG: Index size: 1177392109
[2022-06-10 15:35:29] DEBUG: Mean k-mer index frequency: 16.9463
[2022-06-10 15:35:29] DEBUG: Peak RAM usage: 24 Gb
[2022-06-10 15:35:29] DEBUG: Estimating k-mer identity bias
[2022-06-10 15:36:07] DEBUG: Initial divergence estimate : 0.0935919
[2022-06-10 15:36:07] DEBUG: Relative threshold: Y
[2022-06-10 15:36:07] DEBUG: Max divergence threshold set to 0.193592
[2022-06-10 15:36:07] INFO: Extending reads
[2022-06-10 15:36:07] DEBUG: Estimating overlap coverage
[2022-06-10 15:40:10] INFO: Overlap-based coverage: 2
[2022-06-10 15:40:10] INFO: Median overlap divergence: 0.0949632
[2022-06-10 15:40:10] DEBUG: Sequence divergence distribution:
| * |
| * * |
| *** |
| *** |
| *** |
| * ***** |
| ******* |
| ******** |
| ******** |
| ********** |
| *********** |
| *********** |
| *********** |
| ************** |
| ***************** |
| ***************** |
| ****************** * |
| ********************** ** |
| ***************************** *| * *
| **************************************** *********** * * ** **** ** *
----------------------------------------------------------------------------------------------------
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
Q25 = 0.079, Q50 = 0.095, Q75 = 0.12
[2022-06-10 15:40:11] DEBUG: Assembled disjointig 1
With 2 reads
Start read: +656630b4-ddb7-4dbe-b7a3-0842c611b729
At position: 1
leftTip: 1 rightTip: 1
Suspicious: 1
Short ext: 1
Mean extensions: 1
Avg overlap len: 3312
Min overlap len: 3312
Inner reads: 0
Length: 13593
[2022-06-10 15:40:11] DEBUG: Inner: 6 covered: 50 total: 868712
...
[2022-06-10 20:37:54] DEBUG: Assembled disjointig 44143
With 4 reads
Start read: +74589440-38af-47d4-b759-f37320e104a6
At position: 3
leftTip: 0 rightTip: 0
Suspicious: 2
Short ext: 2
Mean extensions: 4
Avg overlap len: 4763
Min overlap len: 1214
Inner reads: 0
Length: 20675
[2022-06-10 20:37:54] DEBUG: Inner: 544386 covered: 803906 total: 868712
[2022-06-10 20:37:57] INFO: Assembled 44143 disjointigs
[2022-06-10 20:37:58] INFO: Generating sequence
[2022-06-10 20:38:15] DEBUG: Building positional index
[2022-06-10 20:38:15] DEBUG: Total sequence: 1148840534 bp
[2022-06-10 20:39:01] DEBUG: Mean k-mer frequency: 2.80929
[2022-06-10 20:39:01] DEBUG: Repetitive k-mer frequency: 280
[2022-06-10 20:39:01] DEBUG: Filtered 177019530 repetitive k-mers (0.154186)
[2022-06-10 20:39:45] DEBUG: Sorting k-mer index
[2022-06-10 20:39:53] DEBUG: Selected k-mers: 408506424
[2022-06-10 20:39:53] DEBUG: K-mer index size: 971070573
[2022-06-10 20:39:53] DEBUG: Mean k-mer frequency: 2.37712
[2022-06-10 20:39:53] DEBUG: Minimizer rate: 1.18307
[2022-06-10 20:39:53] INFO: Filtering contained disjointigs
[2022-06-10 20:50:15] DEBUG: Computing transitive closure for overlaps
[2022-06-10 20:50:16] DEBUG: Found 1001808 overlaps
[2022-06-10 20:50:16] DEBUG: Left 412352 overlaps after filtering
[2022-06-10 20:50:16] INFO: Contained seqs: 2387
[2022-06-10 20:50:18] DEBUG: Writing FASTA
[2022-06-10 20:50:25] DEBUG: Peak RAM usage: 31 Gb
-----------End assembly log------------
[2022-06-10 20:50:31] root: DEBUG: Disjointigs length: 1110222351, N50: 35360
[2022-06-10 20:50:31] root: INFO: >>>STAGE: consensus
[2022-06-10 20:50:31] root: INFO: Running Minimap2
[2022-06-10 21:32:11] root: ERROR: Error running minimap2, terminating. See the alignment error log for details: /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/10-consensus/minimap.stderr
[2022-06-10 21:32:11] root: ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 12 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/10-consensus/sort_220610_205031' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial13/10-consensus/minimap.bam'"]' returned non-zero exit status 1.
[2022-06-10 21:32:11] root: ERROR: Pipeline aborted
The difference to before, is we now have : non-zero exit status 1, not 37.
For fun, and to get more reads, I then ran -m 2500; which goes back to non-zero exit status 37. The readout:
[2022-06-13 15:17:25] root: INFO: Starting Flye 2.9-b1768
[2022-06-13 15:17:25] root: DEBUG: Cmd: /home/jhowie/miniconda3/bin/flye --nano-raw 1_Cmel4_1_2_PCR1-2-3_SUP.fastq --out-dir /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/ --genome-size 438m --asm-coverage 50 -m 2500 --threads 24 --iterations 1
[2022-06-13 15:17:25] root: DEBUG: Python version: 3.8.5 (default, Sep 4 2020, 07:30:14)
[GCC 7.3.0]
[2022-06-13 15:17:25] root: INFO: >>>STAGE: configure
[2022-06-13 15:17:25] root: INFO: Configuring run
[2022-06-13 15:19:34] root: INFO: Total read length: 23150251229
[2022-06-13 15:19:34] root: INFO: Input genome size: 438000000
[2022-06-13 15:19:34] root: INFO: Estimated coverage: 52
[2022-06-13 15:19:34] root: INFO: Reads N50/N90: 1746 / 566
[2022-06-13 15:19:34] root: INFO: Selected minimum overlap: 2500
[2022-06-13 15:19:34] root: INFO: Using longest 50x reads for contig assembly
[2022-06-13 15:19:39] root: DEBUG: Min read length cutoff: 444
[2022-06-13 15:19:39] root: INFO: >>>STAGE: assembly
[2022-06-13 15:19:39] root: INFO: Assembling disjointigs
[2022-06-13 15:19:39] root: DEBUG: -----Begin assembly log------
[2022-06-13 15:19:39] root: DEBUG: Running: flye-modules assemble --reads /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq --out-asm /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/00-assembly/draft_assembly.fasta --config /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_raw_reads.cfg --log /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/flye.log --threads 24 --genome-size 438000000 --min-ovlp 2500 --min-read 444
[2022-06-13 15:19:39] DEBUG: Build date: Feb 22 2022 03:49:40
[2022-06-13 15:19:39] DEBUG: Total RAM: 125 Gb
[2022-06-13 15:19:39] DEBUG: Available RAM: 121 Gb
[2022-06-13 15:19:39] DEBUG: Total CPUs: 48
[2022-06-13 15:19:39] DEBUG: Loading /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_raw_reads.cfg
[2022-06-13 15:19:39] DEBUG: Loading /home/jhowie/miniconda3/lib/python3.8/site-packages/flye/config/bin_cfg/asm_defaults.cfg
[2022-06-13 15:19:39] DEBUG: big_genome_threshold=29000000
[2022-06-13 15:19:39] DEBUG: meta_read_filter_kmer_freq=100
[2022-06-13 15:19:39] DEBUG: chain_large_gap_penalty=2
[2022-06-13 15:19:39] DEBUG: chain_small_gap_penalty=0.5
[2022-06-13 15:19:39] DEBUG: chain_gap_jump_threshold=100
[2022-06-13 15:19:39] DEBUG: max_coverage_drop_rate=5
[2022-06-13 15:19:39] DEBUG: max_extensions_drop_rate=5
[2022-06-13 15:19:39] DEBUG: chimera_window=100
[2022-06-13 15:19:39] DEBUG: chimera_overhang=1000
[2022-06-13 15:19:39] DEBUG: min_reads_in_disjointig=4
[2022-06-13 15:19:39] DEBUG: max_inner_reads=10
[2022-06-13 15:19:39] DEBUG: max_inner_fraction=0.25
[2022-06-13 15:19:39] DEBUG: max_separation=500
[2022-06-13 15:19:39] DEBUG: unique_edge_length=50000
[2022-06-13 15:19:39] DEBUG: min_repeat_res_support=0.51
[2022-06-13 15:19:39] DEBUG: out_paths_ratio=5
[2022-06-13 15:19:39] DEBUG: graph_cov_drop_rate=5
[2022-06-13 15:19:39] DEBUG: coverage_estimate_window=100
[2022-06-13 15:19:39] DEBUG: max_bubble_length=50000
[2022-06-13 15:19:39] DEBUG: loop_coverage_rate=1.5
[2022-06-13 15:19:39] DEBUG: repeat_edge_cov_mult=1.75
[2022-06-13 15:19:39] DEBUG: weak_detach_rate=5
[2022-06-13 15:19:39] DEBUG: tip_coverage_rate=2
[2022-06-13 15:19:39] DEBUG: tip_length_rate=2
[2022-06-13 15:19:39] DEBUG: output_gfa_before_rr=0
[2022-06-13 15:19:39] DEBUG: low_cutoff_warning=1
[2022-06-13 15:19:39] DEBUG: kmer_size=17
[2022-06-13 15:19:39] DEBUG: use_minimizers=0
[2022-06-13 15:19:39] DEBUG: reads_base_alignment=0
[2022-06-13 15:19:39] DEBUG: meta_read_top_kmer_rate=0.40
[2022-06-13 15:19:39] DEBUG: maximum_jump=1500
[2022-06-13 15:19:39] DEBUG: maximum_overhang=1500
[2022-06-13 15:19:39] DEBUG: repeat_kmer_rate=100
[2022-06-13 15:19:39] DEBUG: assemble_ovlp_divergence=0.10
[2022-06-13 15:19:39] DEBUG: assemble_divergence_relative=1
[2022-06-13 15:19:39] DEBUG: repeat_graph_ovlp_divergence=0.08
[2022-06-13 15:19:39] DEBUG: read_align_ovlp_divergence=0.25
[2022-06-13 15:19:39] DEBUG: hpc_scoring_on=0
[2022-06-13 15:19:39] DEBUG: add_unassembled_reads=0
[2022-06-13 15:19:39] DEBUG: extend_contigs_with_repeats=0
[2022-06-13 15:19:39] DEBUG: min_read_cov_cutoff=3
[2022-06-13 15:19:39] DEBUG: short_tip_length=20000
[2022-06-13 15:19:39] DEBUG: long_tip_length=100000
[2022-06-13 15:19:39] DEBUG: Running with k-mer size: 17
[2022-06-13 15:19:39] DEBUG: Running with minimum overlap 2500
[2022-06-13 15:19:39] DEBUG: Metagenome mode: N
[2022-06-13 15:19:39] DEBUG: Short mode: N
[2022-06-13 15:19:39] INFO: Reading sequences
[2022-06-13 15:21:01] DEBUG: Building positional index
[2022-06-13 15:21:04] DEBUG: Total sequence: 8418638046 bp
[2022-06-13 15:21:05] INFO: Counting k-mers:
[2022-06-13 15:22:40] DEBUG: Updating k-mer histogram
[2022-06-13 15:24:09] DEBUG: Hash size: 65888693
[2022-06-13 15:24:09] DEBUG: Total k-mers 1417619498
[2022-06-13 15:24:10] INFO: Filling index table (1/2)
[2022-06-13 15:27:02] DEBUG: Mean k-mer frequency: 47.0223
[2022-06-13 15:27:02] DEBUG: Repetitive k-mer frequency: 4702
[2022-06-13 15:27:02] DEBUG: Filtered 950381572 repetitive k-mers (0.286516)
[2022-06-13 15:27:06] INFO: Filling index table (2/2)
[2022-06-13 15:30:03] DEBUG: Sorting k-mer index
[2022-06-13 15:30:23] DEBUG: Selected k-mers: 89084835
[2022-06-13 15:30:23] DEBUG: Index size: 2371221948
[2022-06-13 15:30:23] DEBUG: Mean k-mer index frequency: 26.6176
[2022-06-13 15:30:23] DEBUG: Peak RAM usage: 36 Gb
[2022-06-13 15:30:23] DEBUG: Estimating k-mer identity bias
[2022-06-13 15:30:35] DEBUG: Initial divergence estimate : 0.0942718
[2022-06-13 15:30:35] DEBUG: Relative threshold: Y
[2022-06-13 15:30:35] DEBUG: Max divergence threshold set to 0.194272
[2022-06-13 15:30:35] INFO: Extending reads
[2022-06-13 15:30:35] INFO: Extending reads
[2022-06-13 15:30:35] DEBUG: Estimating overlap coverage
[2022-06-13 15:33:26] INFO: Overlap-based coverage: 6
[2022-06-13 15:33:26] INFO: Median overlap divergence: 0.0968613
[2022-06-13 15:33:26] DEBUG: Sequence divergence distribution:
| * |
| ** |
| ** * |
| **** * |
| ****** * |
| ******** |
| * ******** |
| * ********* |
| *********** |
| *********** |
| ************ |
| ************* |
| ************* |
| **************** |
| ***************** * |
| ********************* * |
| ********************* ** |
| * ************************ * | *
| * *********************************|* ******** ******** * * *
----------------------------------------------------------------------------------------------------
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
Q25 = 0.079, Q50 = 0.097, Q75 = 0.12
[2022-06-13 15:33:27] DEBUG: Assembled disjointig 1
With 1 reads
Start read: +90891108-1264-4a41-a871-d0bdb7bd835c
At position: 1
leftTip: 1 rightTip: 1
Suspicious: 0
Short ext: 0
Mean extensions: 2
Avg overlap len: 0
Min overlap len: 0
Inner reads: 0
Length: 2681
[2022-06-13 15:33:27] DEBUG: Inner: 2 covered: 64 total: 3548400
...
[2022-06-13 19:45:42] DEBUG: Assembled disjointig 80208
With 3 reads
Start read: +77b4ed94-3970-485d-92c1-770650965ba0
At position: 2
leftTip: 0 rightTip: 0
Suspicious: 2
Short ext: 2
Mean extensions: 288
Avg overlap len: 1643
Min overlap len: 1643
Inner reads: 0
Length: 11092
[2022-06-13 19:45:42] DEBUG: Inner: 2821538 covered: 2692332 total: 3548400
[2022-06-13 19:46:29] INFO: Assembled 80208 disjointigs
[2022-06-13 19:46:31] INFO: Generating sequence
[2022-06-13 19:46:49] DEBUG: Building positional index
[2022-06-13 19:46:49] DEBUG: Total sequence: 1500542806 bp
[2022-06-13 19:47:26] DEBUG: Mean k-mer frequency: 3.20001
[2022-06-13 19:47:26] DEBUG: Repetitive k-mer frequency: 320
[2022-06-13 19:47:26] DEBUG: Filtered 256470176 repetitive k-mers (0.171074)
[2022-06-13 19:47:58] DEBUG: Sorting k-mer index
[2022-06-13 19:48:08] DEBUG: Selected k-mers: 468286810
[2022-06-13 19:48:08] DEBUG: K-mer index size: 1242709094
[2022-06-13 19:48:08] DEBUG: Mean k-mer frequency: 2.65373
[2022-06-13 19:48:08] DEBUG: Minimizer rate: 1.20748
[2022-06-13 19:48:08] INFO: Filtering contained disjointigs
[2022-06-13 19:54:55] DEBUG: Computing transitive closure for overlaps
[2022-06-13 19:54:56] DEBUG: Found 1555226 overlaps
[2022-06-13 19:54:56] DEBUG: Left 651144 overlaps after filtering
[2022-06-13 19:54:56] INFO: Contained seqs: 9225
[2022-06-13 19:54:59] DEBUG: Writing FASTA
[2022-06-13 19:55:06] DEBUG: Peak RAM usage: 41 Gb
-----------End assembly log------------
[2022-06-13 19:55:16] root: DEBUG: Disjointigs length: 1410083426, N50: 28321
[2022-06-13 19:55:16] root: INFO: >>>STAGE: consensus
[2022-06-13 19:55:16] root: INFO: Running Minimap2
[2022-06-13 20:13:34] root: ERROR: Error running minimap2, terminating. See the alignment error log for details: /mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/10-consensus/minimap.stderr
[2022-06-13 20:13:34] root: ERROR: Command '['/bin/bash', '-c', "set -eo pipefail; flye-minimap2 '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/00-assembly/draft_assembly.fasta' '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/1_Cmel4_1_2_PCR1-2-3_SUP.fastq' -x map-ont -t 24 -a -p 0.5 -N 10 --sam-hit-only -L -K 5G -z 1000 -Q --secondary-seq -I 64G | flye-samtools view -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/00-assembly/draft_assembly.fasta' -u - | flye-samtools sort -T '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/10-consensus/sort_220613_195516' -O bam -@ 4 -l 1 -m 4G -o '/mnt/raid10/cacopsylla/jhowie/3_Nanopore_DraftGenomes/7_Cmel4_1_2_PCR1-2-3_Guppy_SUP_Q10_Canu/Cmel4_FlyeTrial14/10-consensus/minimap.bam'"]' returned non-zero exit status 137.
[2022-06-13 20:13:34] root: ERROR: Pipeline aborted
It seems it is a memory issue at the minimap2 step. Nonetheless, if you have any other ideas, I'd be happy to hear. From my side, a couple of thoughts:
- would using corrected reads from Canu as input help, perhaps [as less work for minimap2]?
- is there a way /is it advisable to take the disjointig fasta, and polish that in another assembler, or would that not make sense?
As a tangential question:
I ran -m 5000, then -m 2500; I also tried for nano-raw and nano-hq; the results are:
--nano-hq -m 5000: Result = N50 33,762, 1172292954 bp = 2.67x inflated; 46,831 disjointigs --nano-raw -m 5000: Result = N50 33,630, 1110222351 bp = 2.53x inflated; 44,143 disjointigs
--nano-raw -m 2500: Result = N50 28,321, 1410083426 bp = 3.22x inflated; 80,208 disjointigs --nano-hq -m 2500: Result = N50 24,367, 1721925195 bp = 3.93x inflated; 109,982 disjointigs
From this I saw, that the switch to nano-raw gives similar N50, but fewer disjointigs and less inflated genome at this step. If I switch to -m 2500, in both cases, the N50 reduces, and the disjointig count and genome inflantion increases, at this step.
From this, I have a question:
-
are the highly inflated genome sizes because of the repeats, or heterozygosity [the individual is not 100% inbred], or is it just what happens at this stage in the pipe? Would I expect this to go away later in the Flye pipe, if I can get it to work?
-
Also, does the longer - fewer contigs on -m 5000, suggest that would be the way to go, or is that irrelevant when focused on just step one of Flye, given later polishing if I can get it to run?
Thanks for any help :)
@JamesHowie14 thanks for the update. I think it could be both a lot of repeats and high heterozygosity. Both a very complicated to assemble. I don't think error-corrected reads will make much difference, but maybe worth trying. Disjointigs may contain duplications and errors, so they should not be used for downstream analysis.
The best solution would really be trying to find a machine with more memory and let minimap2 finish.. One other thing to try running purge_haplotigs (https://bitbucket.org/mroachawri/purge_haplotigs/src/master/) on consensus.fasta, which should collapse the duplications. This at least will give an idea how much duplicated sequence is there.
Dear fenderglass, that sounds like good advice to me. I'll definitely try out both! Thanks :)
Closed due to inactivity, feel free to reopen if there are any follow-ups.