canu
canu copied to clipboard
Published
20 hours ago •
marbl
marbl
HiCanu failing for real world HiFi data
I am trying to use HiCanu on some real world HiFi data that I downloaded from NCBI but for both the datasets it is not giving me any output. I have already used HiCanu for my simulated data (all with 10x coverage).
This is the command I used for running the real world data: ~/bin/canu -assemble -p Nibea -d /home/CanuOutputs/Nibea/ genomeSize=615m -pacbio-hifi /home/SRAFiles/NibeaCoibor.fa
This is my job.e log file:
canu 2.2
CONFIGURE CANU
Detected Java(TM) Runtime Environment '1.8.0_272' (from 'java') with -d64 support.
Detected gnuplot version '4.2 patchlevel 6 ' (from 'gnuplot') and image format 'png'.
Detected 1 CPUs and 252 gigabytes of memory on the local machine.
Detected PBS/Torque '5.1.3' with 'pbsnodes' binary in /usr/bin/pbsnodes.
Detecting PBS/Torque resources.
PBS/Torque support detected. Resources available:
1 host with 64 cores and 252 GB memory.
1 host with 72 cores and 252 GB memory.
6 hosts with 64 cores and 126 GB memory.
1 host with 24 cores and 252 GB memory.
6 hosts with 72 cores and 125 GB memory.
2 hosts with 40 cores and 252 GB memory.
(tag)Threads
(tag)Memory |
(tag) | | algorithm
------- ---------- -------- -----------------------------
Grid: meryl 13.000 GB 8 CPUs (k-mer counting)
Grid: hap 12.000 GB 8 CPUs (read-to-haplotype assignment)
Grid: cormhap 20.000 GB 12 CPUs (overlap detection with mhap)
Grid: obtovl 16.000 GB 12 CPUs (overlap detection)
Grid: utgovl 16.000 GB 12 CPUs (overlap detection)
Grid: cor -.--- GB 4 CPUs (read correction)
Grid: ovb 4.000 GB 1 CPU (overlap store bucketizer)
Grid: ovs 16.000 GB 1 CPU (overlap store sorting)
Grid: red 17.000 GB 8 CPUs (read error detection)
Grid: oea 8.000 GB 1 CPU (overlap error adjustment)
Grid: bat 252.000 GB 16 CPUs (contig construction with bogart)
Grid: cns -.--- GB 8 CPUs (consensus)
Found trimmed raw PacBio HiFi reads in the input files.
Generating assembly 'Nibea' in '/home/CanuOutputs/Nibea':
genomeSize:
615000000
Overlap Generation Limits:
corOvlErrorRate 0.0000 ( 0.00%)
obtOvlErrorRate 0.0250 ( 2.50%)
utgOvlErrorRate 0.0100 ( 1.00%)
Overlap Processing Limits:
corErrorRate 0.0000 ( 0.00%)
obtErrorRate 0.0250 ( 2.50%)
utgErrorRate 0.0003 ( 0.03%)
cnsErrorRate 0.0500 ( 5.00%)
Stages to run:
assemble HiFi reads.
Correction skipped; not enabled.
Trimming skipped; not enabled.
BEGIN ASSEMBLY
Starting command on Fri Jun 21 13:14:39 2024 with 45186.208 GB free disk space
cd .
./Nibea.seqStore.sh \
> ./Nibea.seqStore.err 2>&1
Finished on Fri Jun 21 13:21:09 2024 (390 seconds) with 45180.14 GB free disk space
In sequence store './Nibea.seqStore':
Found 1919461 reads.
Found 29259679568 bases (47.57 times coverage).
Histogram of corrected reads:
G=29259679568 sum of || length num
NG length index lengths || range seqs
00010 19729 137059 2925977360 || 9956-10551 12111|-----
00020 18158 292134 5851941183 || 10552-11147 24390|---------
00030 17056 458593 8777910356 || 11148-11743 71448|-------------------------
00040 16148 634993 11703878635 || 11744-12339 137671|------------------------------------------------
00050 15335 820976 14629854418 || 12340-12935 178710|---------------------------------------------------------------
00060 14576 1016711 17555807793 || 12936-13531 181045|---------------------------------------------------------------
00070 13842 1222700 20481784186 || 13532-14127 173229|-------------------------------------------------------------
00080 13115 1439830 23407745673 || 14128-14723 163730|---------------------------------------------------------
00090 12355 1669496 26333723186 || 14724-15319 152301|-----------------------------------------------------
00100 9956 1919460 29259679568 || 15320-15915 139169|-------------------------------------------------
001.000x 1919461 29259679568 || 15916-16511 125638|--------------------------------------------
|| 16512-17107 110391|---------------------------------------
|| 17108-17703 95375|----------------------------------
|| 17704-18299 79796|----------------------------
|| 18300-18895 66488|------------------------
|| 18896-19491 52927|-------------------
|| 19492-20087 41922|---------------
|| 20088-20683 32421|------------
|| 20684-21279 24621|---------
|| 21280-21875 18105|-------
|| 21876-22471 12807|-----
|| 22472-23067 8902|----
|| 23068-23663 6108|---
|| 23664-24259 3847|--
|| 24260-24855 2393|-
|| 24856-25451 1504|-
|| 25452-26047 930|-
|| 26048-26643 536|-
|| 26644-27239 313|-
|| 27240-27835 208|-
|| 27836-28431 118|-
|| 28432-29027 66|-
|| 29028-29623 51|-
|| 29624-30219 33|-
|| 30220-30815 29|-
|| 30816-31411 25|-
|| 31412-32007 25|-
|| 32008-32603 12|-
|| 32604-33199 24|-
|| 33200-33795 7|-
|| 33796-34391 8|-
|| 34392-34987 2|-
|| 34988-35583 6|-
|| 35584-36179 9|-
|| 36180-36775 1|-
|| 36776-37371 2|-
|| 37372-37967 1|-
|| 37968-38563 4|-
|| 38564-39159 0|
|| 39160-39755 2|-
In sequence store './Nibea.seqStore':
Found 1919461 reads.
Found 29259679568 bases (47.57 times coverage).
Histogram of corrected-trimmed reads:
G=29259679568 sum of || length num
NG length index lengths || range seqs
00010 19729 137059 2925977360 || 9956-10551 12111|-----
00020 18158 292134 5851941183 || 10552-11147 24390|---------
00030 17056 458593 8777910356 || 11148-11743 71448|-------------------------
00040 16148 634993 11703878635 || 11744-12339 137671|------------------------------------------------
00050 15335 820976 14629854418 || 12340-12935 178710|---------------------------------------------------------------
00060 14576 1016711 17555807793 || 12936-13531 181045|---------------------------------------------------------------
00070 13842 1222700 20481784186 || 13532-14127 173229|-------------------------------------------------------------
00080 13115 1439830 23407745673 || 14128-14723 163730|---------------------------------------------------------
00090 12355 1669496 26333723186 || 14724-15319 152301|-----------------------------------------------------
00100 9956 1919460 29259679568 || 15320-15915 139169|-------------------------------------------------
001.000x 1919461 29259679568 || 15916-16511 125638|--------------------------------------------
|| 16512-17107 110391|---------------------------------------
|| 17108-17703 95375|----------------------------------
|| 17704-18299 79796|----------------------------
|| 18300-18895 66488|------------------------
|| 18896-19491 52927|-------------------
|| 19492-20087 41922|---------------
|| 20088-20683 32421|------------
|| 20684-21279 24621|---------
|| 21280-21875 18105|-------
|| 21876-22471 12807|-----
|| 22472-23067 8902|----
|| 23068-23663 6108|---
|| 23664-24259 3847|--
|| 24260-24855 2393|-
|| 24856-25451 1504|-
|| 25452-26047 930|-
|| 26048-26643 536|-
|| 26644-27239 313|-
|| 27240-27835 208|-
|| 27836-28431 118|-
|| 28432-29027 66|-
|| 29028-29623 51|-
|| 29624-30219 33|-
|| 30220-30815 29|-
|| 30816-31411 25|-
|| 31412-32007 25|-
|| 32008-32603 12|-
|| 32604-33199 24|-
|| 33200-33795 7|-
|| 33796-34391 8|-
|| 34392-34987 2|-
|| 34988-35583 6|-
|| 35584-36179 9|-
|| 36180-36775 1|-
|| 36776-37371 2|-
|| 37372-37967 1|-
|| 37968-38563 4|-
|| 38564-39159 0|
|| 39160-39755 2|-
----------------------------------------
Starting command on Fri Jun 21 13:21:25 2024 with 45179.015 GB free disk space
cd unitigging/0-mercounts
./meryl-configure.sh \
> ./meryl-configure.err 2>&1
Finished on Fri Jun 21 13:21:27 2024 (2 seconds) with 45179.015 GB free disk space
segments memory batches
01 11.97 GB 7
02 11.97 GB 4
04 8.91 GB 3
06 11.97 GB 2
08 8.91 GB 2
12 6.37 GB 2
16 4.81 GB 2
20 4.04 GB 2
24 3.36 GB 2
32 2.47 GB 2
40 2.14 GB 2
48 1.78 GB 2
56 1.53 GB 2
64 1.34 GB 2
96 0.90 GB 2
For 1919461 reads with 29259679568 bases, limit to 292 batches.
Will count kmers using 06 jobs, each using 13 GB and 8 threads.
Finished stage 'merylConfigure', reset canuIteration.
Running jobs. First attempt out of 2.
'meryl-count.jobSubmit-01.sh' -> job 1663532[].mgt2-ib.local tasks 1-6.
Starting command on Fri Jun 21 13:21:27 2024 with 45179.015 GB free disk space
cd /home/obhowmik/CanuOutputs/All_LR_Together/Nibea
qsub \
-j oe \
-d `pwd` \
-W depend=afteranyarray:1663532[].mgt2-ib.local \
-l nodes=1:ppn=1,mem=4g \
-N 'canu_Nibea' \
-o canu-scripts/canu.01.out canu-scripts/canu.01.sh
Finished on Fri Jun 21 13:21:27 2024 (fast as lightning) with 45179.015 GB free disk space
----------------------------------------
----------------------------------------
--This is the canuscript which I found in the canuscripts folder:
Found perl:
/usr/bin/perl
Found java:
/usr/bin/java
openjdk version "1.8.0_272"
Found canu:
/home/obhowmik/canu-2.2/bin/canu
canu 2.2
CONFIGURE CANU
Detected Java(TM) Runtime Environment '1.8.0_272' (from 'java') with -d64 support.
Detected gnuplot version '4.2 patchlevel 6 ' (from 'gnuplot') and image format 'png'.
Detected 1 CPUs and 252 gigabytes of memory on the local machine.
Detected PBS/Torque '5.1.3' with 'pbsnodes' binary in /usr/bin/pbsnodes.
Detecting PBS/Torque resources.
PBS/Torque support detected. Resources available:
1 host with 64 cores and 252 GB memory.
1 host with 72 cores and 252 GB memory.
6 hosts with 64 cores and 126 GB memory.
1 host with 24 cores and 252 GB memory.
6 hosts with 72 cores and 125 GB memory.
2 hosts with 40 cores and 252 GB memory.
(tag)Threads
(tag)Memory |
(tag) | | algorithm
Grid: meryl 25.000 GB 8 CPUs (k-mer counting)
Grid: hap 16.000 GB 16 CPUs (read-to-haplotype assignment)
Grid: cormhap 20.000 GB 12 CPUs (overlap detection with mhap)
Grid: obtovl 16.000 GB 12 CPUs (overlap detection)
Grid: utgovl 16.000 GB 12 CPUs (overlap detection)
Grid: cor -.--- GB 4 CPUs (read correction)
Grid: ovb 4.000 GB 1 CPU (overlap store bucketizer)
Grid: ovs 32.000 GB 1 CPU (overlap store sorting)
Grid: red 17.000 GB 8 CPUs (read error detection)
Grid: oea 8.000 GB 1 CPU (overlap error adjustment)
Grid: bat 252.000 GB 16 CPUs (contig construction with bogart)
Grid: cns -.--- GB 8 CPUs (consensus)
--
Found PacBio HiFi reads in 'Nibea.seqStore':
Libraries:
PacBio HiFi: 1
Reads:
Corrected: 29259679568
Corrected and Trimmed: 29259679568
Generating assembly 'Nibea' in '/home/CanuOutputs/Caddisfly':
genomeSize:
1170000000
Overlap Generation Limits:
corOvlErrorRate 0.0000 ( 0.00%)
obtOvlErrorRate 0.0250 ( 2.50%)
utgOvlErrorRate 0.0100 ( 1.00%)
Overlap Processing Limits:
corErrorRate 0.0000 ( 0.00%)
obtErrorRate 0.0250 ( 2.50%)
utgErrorRate 0.0003 ( 0.03%)
cnsErrorRate 0.0500 ( 5.00%)
Stages to run:
assemble HiFi reads.
Correction skipped; not enabled.
--
Trimming skipped; not enabled.
--
BEGIN ASSEMBLY
--
Overlap jobs failed, retry.
job unitigging/1-overlapper/001/000005.ovb FAILED.
job unitigging/1-overlapper/001/000006.ovb FAILED.
job unitigging/1-overlapper/001/000007.ovb FAILED.
job unitigging/1-overlapper/001/000008.ovb FAILED.
job unitigging/1-overlapper/001/000009.ovb FAILED.
job unitigging/1-overlapper/001/000010.ovb FAILED.
job unitigging/1-overlapper/001/000011.ovb FAILED.
job unitigging/1-overlapper/001/000012.ovb FAILED.
job unitigging/1-overlapper/001/000013.ovb FAILED.
job unitigging/1-overlapper/001/000014.ovb FAILED.
job unitigging/1-overlapper/001/000015.ovb FAILED.
job unitigging/1-overlapper/001/000016.ovb FAILED.
job unitigging/1-overlapper/001/000017.ovb FAILED.
job unitigging/1-overlapper/001/000018.ovb FAILED.
job unitigging/1-overlapper/001/000019.ovb FAILED.
job unitigging/1-overlapper/001/000020.ovb FAILED.
job unitigging/1-overlapper/001/000021.ovb FAILED.
job unitigging/1-overlapper/001/000022.ovb FAILED.
job unitigging/1-overlapper/001/000023.ovb FAILED.
job unitigging/1-overlapper/001/000024.ovb FAILED.
job unitigging/1-overlapper/001/000025.ovb FAILED.
job unitigging/1-overlapper/001/000026.ovb FAILED.
job unitigging/1-overlapper/001/000027.ovb FAILED.
job unitigging/1-overlapper/001/000028.ovb FAILED.
job unitigging/1-overlapper/001/000029.ovb FAILED.
job unitigging/1-overlapper/001/000030.ovb FAILED.
job unitigging/1-overlapper/001/000031.ovb FAILED.
job unitigging/1-overlapper/001/000032.ovb FAILED.
job unitigging/1-overlapper/001/000033.ovb FAILED.
job unitigging/1-overlapper/001/000034.ovb FAILED.
job unitigging/1-overlapper/001/000035.ovb FAILED.
job unitigging/1-overlapper/001/000036.ovb FAILED.
job unitigging/1-overlapper/001/000037.ovb FAILED.
job unitigging/1-overlapper/001/000038.ovb FAILED.
job unitigging/1-overlapper/001/000039.ovb FAILED.
job unitigging/1-overlapper/001/000040.ovb FAILED.
job unitigging/1-overlapper/001/000041.ovb FAILED.
job unitigging/1-overlapper/001/000042.ovb FAILED.
job unitigging/1-overlapper/001/000043.ovb FAILED.
job unitigging/1-overlapper/001/000044.ovb FAILED.
job unitigging/1-overlapper/001/000045.ovb FAILED.
job unitigging/1-overlapper/001/000046.ovb FAILED.
job unitigging/1-overlapper/001/000047.ovb FAILED.
job unitigging/1-overlapper/001/000049.ovb FAILED.
job unitigging/1-overlapper/001/000051.ovb FAILED.
job unitigging/1-overlapper/001/000052.ovb FAILED.
job unitigging/1-overlapper/001/000053.ovb FAILED.
job unitigging/1-overlapper/001/000055.ovb FAILED.
job unitigging/1-overlapper/001/000056.ovb FAILED.
job unitigging/1-overlapper/001/000057.ovb FAILED.
job unitigging/1-overlapper/001/000058.ovb FAILED.
job unitigging/1-overlapper/001/000059.ovb FAILED.
job unitigging/1-overlapper/001/000060.ovb FAILED.
job unitigging/1-overlapper/001/000061.ovb FAILED.
job unitigging/1-overlapper/001/000062.ovb FAILED.
job unitigging/1-overlapper/001/000063.ovb FAILED.
job unitigging/1-overlapper/001/000064.ovb FAILED.
job unitigging/1-overlapper/001/000065.ovb FAILED.
job unitigging/1-overlapper/001/000066.ovb FAILED.
job unitigging/1-overlapper/001/000067.ovb FAILED.
job unitigging/1-overlapper/001/000068.ovb FAILED.
job unitigging/1-overlapper/001/000069.ovb FAILED.
job unitigging/1-overlapper/001/000070.ovb FAILED.
job unitigging/1-overlapper/001/000071.ovb FAILED.
job unitigging/1-overlapper/001/000072.ovb FAILED.
job unitigging/1-overlapper/001/000073.ovb FAILED.
job unitigging/1-overlapper/001/000074.ovb FAILED.
job unitigging/1-overlapper/001/000075.ovb FAILED.
job unitigging/1-overlapper/001/000076.ovb FAILED.
job unitigging/1-overlapper/001/000077.ovb FAILED.
job unitigging/1-overlapper/001/000078.ovb FAILED.
job unitigging/1-overlapper/001/000079.ovb FAILED.
job unitigging/1-overlapper/001/000080.ovb FAILED.
job unitigging/1-overlapper/001/000081.ovb FAILED.
job unitigging/1-overlapper/001/000082.ovb FAILED.
job unitigging/1-overlapper/001/000083.ovb FAILED.
job unitigging/1-overlapper/001/000084.ovb FAILED.
job unitigging/1-overlapper/001/000085.ovb FAILED.
job unitigging/1-overlapper/001/000086.ovb FAILED.
job unitigging/1-overlapper/001/000087.ovb FAILED.
job unitigging/1-overlapper/001/000088.ovb FAILED.
Running jobs. Second attempt out of 2.
'overlap.jobSubmit-01.sh' -> job 1664060[].mgt2-ib.local tasks 5-47.
'overlap.jobSubmit-02.sh' -> job 1664061.mgt2-ib.local task 49.
'overlap.jobSubmit-03.sh' -> job 1664062[].mgt2-ib.local tasks 51-53.
'overlap.jobSubmit-04.sh' -> job 1664063[].mgt2-ib.local tasks 55-88.
Starting command on Mon Jun 24 14:45:17 2024 with 45138.439 GB free disk space
cd /home/CanuOutputs/Caddisfly
qsub \
-j oe \
-d `pwd` \
-W depend=afteranyarray:1664060[].mgt2-ib.local:1664061.mgt2-ib.local:1664062[].mgt2-ib.local:1664063[].mgt2-ib.local \
-l nodes=1:ppn=1,mem=5g \
-N 'canu_Nibea' \
-o canu-scripts/canu.04.out canu-scripts/canu.04.sh
qsub: submit error (Invalid Job Dependency)
Finished on Mon Jun 24 14:45:18 2024 (one second) with 45138.439 GB free space
ERROR:
ERROR: Failed with exit code 208. (rc=53248)
ERROR:
Failed to submit Canu executive. Delay 10 seconds and try again.
Starting command on Mon Jun 24 14:45:28 2024 with 45138.439 GB free disk space
cd /home/CanuOutputs/Caddisfly
qsub \
-j oe \
-d `pwd` \
-W depend=afteranyarray:1664060[].mgt2-ib.local:1664061.mgt2-ib.local:1664062[].mgt2-ib.local:1664063[].mgt2-ib.local \
-l nodes=1:ppn=1,mem=5g \
-N 'canu_Nibea' \
-o canu-scripts/canu.04.out canu-scripts/canu.04.sh
qsub: submit error (Invalid Job Dependency)
Finished on Mon Jun 24 14:45:29 2024 (one second) with 45138.439 GB free disk space
ERROR:
ERROR: Failed with exit code 208. (rc=53248)
ERROR:
Failed to submit Canu executive. Giving up after two tries.
Is it a memory issue or something else?
