nanoporetech
Dorado 0.5.3 - memory issues - BAM truncated?
Issue Report
We have some data that I am not able to perform the basecalling. I tried with two approach, but I got different errors in both test.
First command (passing all the pod5 reads)
/dorado-0.5.3-linux-x64/bin/dorado duplex \
/dorado-0.5.3-linux-x64/models/[email protected] \
$INPUT \
-v -r -x "cuda:all" --min-qscore 10 --modified-bases 5mC_5hmC \
> [email protected] 2> [email protected]
First issue
//
...
[2024-03-10 12:30:23.590] [debug] Sorted channel 1212
Killed
//
Second command (two jobs jobs, one for pod5 fail reads and one for pod5 pass reads)
/dorado-0.5.3-linux-x64/bin/dorado duplex \
/dorado-0.5.3-linux-x64/models/[email protected] \
./XXX_rebasecalling/methylation_basecalling_240123_5mC_5hmC-ChannelTable/pod5_pass/split_by_channel_pod5_pass/ \
--min-qscore 10 --modified-bases 5mC_5hmC \
> [email protected]_pod5pass.bam 2> [email protected]_pod5pass.log
Second issue
fail reads
//
...
[2024-04-19 09:41:19.214] [info] > Starting Stereo Duplex pipeline
[2024-04-19 09:41:19.233] [info] > Reading read channel info
[2024-04-19 09:41:34.020] [info] > Processed read channel info
**[2024-04-19 23:58:25.388] [error] basic_string::substr: __pos (which is 7411) > this->size() (which is 5817)**
[2024-04-21 10:22:34.399] [info] > Simplex reads basecalled: 470223
[2024-04-21 10:22:34.399] [info] > Simplex reads filtered: 1908745
[2024-04-21 10:22:34.399] [info] > Duplex reads basecalled: 28201
[2024-04-21 10:22:34.399] [info] > Duplex reads filtered: 10628
[2024-04-21 10:22:34.400] [info] > Duplex rate: 3.4319797%
[2024-04-21 10:22:34.400] [info] > Basecalled @ Bases/s: 1.218011e+05
//
pass reads
//
..
[2024-04-19 09:33:52.687] [info] > No duplex pairs file provided, pairing will be performed automatically
[2024-04-19 09:37:10.935] [info] - set batch size for cuda:0 to 640
[2024-04-19 09:37:19.733] [info] - set batch size for cuda:0 to 1792
[2024-04-19 09:37:19.733] [info] > Starting Stereo Duplex pipeline
[2024-04-19 09:37:19.763] [info] > Reading read channel info
[2024-04-19 09:40:27.087] [info] > Processed read channel info
**[2024-04-23 02:56:05.933] [error] basic_string::substr: __pos (which is 11734) > this->size() (which is 11684)**
//
BAM file
For fail pod5 job is fine, but for the pass pod5 job the BAM file seems truncated.
samtools view [email protected]_pod5pass.bam | most
[W::bam_hdr_read] EOF marker is absent. The input is probably truncated
Please, could you teld me how can we solve?
Many thanks in advance! Cristina
Hi @HalfPhoton, no, we use a local computer. Did you have a similar issue?
No, I'm trying to understand why this issue is happening - You used $INPUT in your example and I was thinking that this could be related to other issues we've seen with users running on compute clusters.
Can you complete the issue template to give us more information to help you?
Hi @HalPhoton,
No, in the bash script I don't use the var $INPUT, I putted the full PATH.
Can you complete the issue template to give us more information to help you?
dorado version used
dorado: 0.5.3 dorado: v0.6.1
lsb_release -a
No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.2 LTS Release: 22.04 Codename: jammy
nvidia-smi
Wed May 8 13:43:53 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02 Driver Version: 530.30.02 CUDA Version: 12.1 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA RTX A6000 On | 00000000:21:00.0 Off | Off |
| 65% 84C P2 300W / 300W| 44603MiB / 49140MiB | 100% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
More info
Please, let me know if you need more information.
Many thanks in advance! Cristina
Hello @cfrias-cnag does basecalling finish on the pass read set or does dorado crash? It's not clear from your log
Hi Mike,
Sorry, I mean from pass reads.
Log file: “[2024-04-24 10:38:16.693] [info] Running: "duplex" @.***" "/scratch_isilon/groups/pbt/cfrias/ont02/output/240123_BCLLATLAS_240_5208AI_PAU40160_rebasecalling/methylation_basecalling_240123_5mC_5hmC-ChannelTable/pod5_pass/split_by_channel_pod5_pass/" "--min-qscore" "10" "--modified-bases" "5mC_5hmC" [2024-04-24 10:38:16.719] [info] > No duplex pairs file provided, pairing will be performed automatically [2024-04-24 10:41:29.710] [info] cuda:0 using chunk size 9996, batch size 576 [2024-04-24 10:41:34.615] [info] cuda:0 using chunk size 10000, batch size 896 [2024-04-24 10:41:35.562] [info] > Starting Stereo Duplex pipeline [2024-04-24 10:41:35.580] [info] > Reading read channel info [2024-04-24 10:44:38.647] [info] > Processed read channel info [2024-04-26 23:01:43.012] [error] basic_string::substr: __pos (which is 10484) > this->size() (which is 10227)”
I thought that maybe I can try to extract the pod5 basecalled from the uBAM, using pod5_subset and launch again the basecalling with those reads. (in case the error is related to some pod5 read )
What do you think about is?
Many thanks! Cris
De: Mike Vella @.> Enviado el: viernes, 17 de mayo de 2024 12:59 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
You don't often get email from @.@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
Hello @cfrias-cnaghttps://github.com/cfrias-cnag does basecalling finish on the pass read set or does dorado crash? It's not clear from your log
— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2117320263, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOLJMPURVSX24Q57OXDZCXPHFAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJXGMZDAMRWGM. You are receiving this because you were mentioned.Message ID: @.@.>>
Also - can I confirm with you: This job was running for 2.5 days?
[2024-04-24 10:44:38.647] [info] > Processed read channel info
[2024-04-26 23:01:43.012] [error] basic_string::substr: __pos (which is 10484) > this->size() (which is 10227)”
No, the job was running 2 days more, 5 in total.
The uBAM file finished to write on 29/04/2024 (and started on 24/04/2024):
@.***
I found a similar message using the FAIL reads basecalling, but then the log file show that the basecalling continues and said that it finished successfully, and furthermore the uBAM is not truncated.
Thanks! Cris
De: Richard Harris @.> Enviado el: viernes, 17 de mayo de 2024 14:11 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
You don't often get email from @.@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
Also - can I confirm with you: This job was running for 2.5 days?
[2024-04-24 10:44:38.647] [info] > Processed read channel info
[2024-04-26 23:01:43.012] [error] basic_string::substr: __pos (which is 10484) > this->size() (which is 10227)”
— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2117459128, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOLO3SZVODPZ5CBDSKLZCXXWZAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJXGQ2TSMJSHA. You are receiving this because you were mentioned.Message ID: @.@.>>
Hi Richard,
Thanks!
I've just launched the same command adding -vv.
Cris
De: Richard Harris @.> Enviado el: viernes, 17 de mayo de 2024 14:10 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
You don't often get email from @.@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
Could you add -vv to your command to generate a detailed log?
Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2117457184, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOLABSOZ5KPXBEVCF5LZCXXRVAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJXGQ2TOMJYGQ. You are receiving this because you were mentioned.Message ID: @.@.>>
Hi Richard,
The job finished.
The uBAM is truncated.
Please, could you tell me how I can send you the log file ( 19GB )?
I've just launched the same basecalling with last version of dorado v.0.7.0 (dorado-0.7.0-linux-x64). (The previous log is produced by dorado v.0.6.1)
Many thanks in advance! Cristina
From: Maria Cristina Frias Lopez Sent: Friday, May 17, 2024 4:13 PM To: nanoporetech/dorado @.>; nanoporetech/dorado @.> Cc: Mention @.>; Matthew John Ingham @.> Subject: RE: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
Hi Richard,
Thanks!
I’ve just launched the same command adding -vv.
Cris
De: Richard Harris @.> Enviado el: viernes, 17 de mayo de 2024 14:10 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
You don't often get email from @.@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
Could you add -vv to your command to generate a detailed log?
— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2117457184, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOLABSOZ5KPXBEVCF5LZCXXRVAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJXGQ2TOMJYGQ. You are receiving this because you were mentioned.Message ID: @.@.>>
Hi @cfrias-cnag, sorry for the delayed response.
Can you update dorado to 0.7.0 and try on a smaller dataset using --max-reads 10000 to see if the issue persists?
Dear Richard,
I updated the version of dorado to the last one,
But I launched before you wrote so when It finished (if the error persists), I will re-launch with the parameter you suggested me(--max-reads 10000) because I did before you wrote me.
I will let you know how it is going.
Many thanks in advance! Cristina
PD: I’m also looking at the community for any update, following this one: https://github.com/nanoporetech/dorado/issues/855
I have to comment with you, that I got similar errors during the basecalling of the “fail reads” but like the other user it were skipped “alone” (by dorado, I didn’t filer any read) and it finished successfully. But, in my case, the issue is related to the subset of “pass read”, after showed this error, dorado was hung and died without any further messages.
De: Richard Harris @.> Enviado el: jueves, 30 de mayo de 2024 11:42 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
Hi @cfrias-cnaghttps://github.com/cfrias-cnag, sorry for the delayed response. Can you update dorado to 0.7.0 and try on a smaller dataset using --max-reads 10000 to see if the issue persists?
— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2139170677, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNONQEOWA3ZQMA2XPZ7DZE3X6PAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZGE3TANRXG4. You are receiving this because you were mentioned.Message ID: @.@.>>
Dear Richard,
The process with dorado v.0.7.0 died after 7 hours running (11GB of uBAM).
With the previous versions of dorado, the process run about 5 days and was produced a bigger uBAM "truncated" (150GB ).
Let me check adding the parameter you suggested me (--max-reads 10000) .
Regards, Cristina
From: Maria Cristina Frias Lopez Sent: Monday, June 3, 2024 12:40 PM To: nanoporetech/dorado @.>; nanoporetech/dorado @.> Cc: Mention @.>; Matthew John Ingham @.>; Simon Heath @.***> Subject: RE: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
Dear Richard,
I updated the version of dorado to the last one,
But I launched before you wrote so when It finished (if the error persists), I will re-launch with the parameter you suggested me(--max-reads 10000) because I did before you wrote me.
I will let you know how it is going.
Many thanks in advance!
Cristina
PD: I’m also looking at the community for any update, following this one:
https://github.com/nanoporetech/dorado/issues/855
I have to comment with you, that I got similar errors during the basecalling of the “fail reads” but like the other user it were skipped “alone” (by dorado, I didn’t filer any read) and it finished successfully. But, in my case, the issue is related to the subset of “pass read”, after showed this error, dorado was hung and died without any further messages.
De: Richard Harris @.> Enviado el: jueves, 30 de mayo de 2024 11:42 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
Hi @cfrias-cnaghttps://github.com/cfrias-cnag, sorry for the delayed response. Can you update dorado to 0.7.0 and try on a smaller dataset using --max-reads 10000 to see if the issue persists?
— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2139170677, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNONQEOWA3ZQMA2XPZ7DZE3X6PAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZGE3TANRXG4. You are receiving this because you were mentioned.Message ID: @.@.>>
Dear Richard,
Seems that this parameter doesn't exist: --max-reads 10000
[cid:f8779588-78c5-4151-92c8-f840dafffa56]
[cid:8519a5a5-46a2-4294-9561-dbb3bf689bb8]
From: Maria Cristina Frias Lopez @.> Sent: Tuesday, June 4, 2024 11:14 AM To: nanoporetech/dorado @.>; nanoporetech/dorado @.> Cc: Mention @.>; Matthew John Ingham @.>; Simon Heath @.> Subject: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
Dear Richard,
The process with dorado v.0.7.0 died after 7 hours running (11GB of uBAM).
With the previous versions of dorado, the process run about 5 days and was produced a bigger uBAM "truncated" (150GB ).
Let me check adding the parameter you suggested me (--max-reads 10000) .
Regards, Cristina
From: Maria Cristina Frias Lopez Sent: Monday, June 3, 2024 12:40 PM To: nanoporetech/dorado @.>; nanoporetech/dorado @.> Cc: Mention @.>; Matthew John Ingham @.>; Simon Heath @.***> Subject: RE: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
Dear Richard,
I updated the version of dorado to the last one,
But I launched before you wrote so when It finished (if the error persists), I will re-launch with the parameter you suggested me(--max-reads 10000) because I did before you wrote me.
I will let you know how it is going.
Many thanks in advance!
Cristina
PD: I’m also looking at the community for any update, following this one:
https://github.com/nanoporetech/dorado/issues/855
I have to comment with you, that I got similar errors during the basecalling of the “fail reads” but like the other user it were skipped “alone” (by dorado, I didn’t filer any read) and it finished successfully. But, in my case, the issue is related to the subset of “pass read”, after showed this error, dorado was hung and died without any further messages.
De: Richard Harris @.> Enviado el: jueves, 30 de mayo de 2024 11:42 Para: nanoporetech/dorado @.> CC: Maria Cristina Frias Lopez @.>; Mention @.> Asunto: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
Hi @cfrias-cnaghttps://github.com/cfrias-cnag, sorry for the delayed response. Can you update dorado to 0.7.0 and try on a smaller dataset using --max-reads 10000 to see if the issue persists?
— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2139170677, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNONQEOWA3ZQMA2XPZ7DZE3X6PAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZZGE3TANRXG4. You are receiving this because you were mentioned.Message ID: @.@.>>
https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/basecaller.cpp#L473
https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/demux.cpp#L106
--max-reads should be available in both simplex and duplex basecalling. Can you share your full command again?
Hi Richard,
I am using "dorado duplex", not "dorado basecaller" neither "dorado demux" (the links you send me).
Here you have the full command:
/home/cfrias/software/dorado-0.7.0-linux-x64/bin/dorado duplex
@.***
/scratch_isilon/groups/pbt/cfrias/ont02/<INPUT>
--min-qscore 10 --modified-bases 5mC_5hmC
--max-reads 10000
-vv > @._pod5pass.dorado-0.7.0-maxreads_10000.bam 2> @._pod5pass.dorado-0.7.0-maxreads_10000.log
I also installed the last version, but I can not see this parameter in the "duplex", just in the "simplex":
duplex
[cid:c514db7b-7515-4a82-9d86-a25dd680d7d1]
simplex
[cid:194c69a2-1bfa-462d-8aec-fc76017a6116]
Many thanks in advance,
Cris
From: Richard Harris @.> Sent: Tuesday, June 4, 2024 2:39 PM To: nanoporetech/dorado @.> Cc: Maria Cristina Frias Lopez @.>; Mention @.> Subject: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/basecaller.cpp#L473
https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/demux.cpp#L106
--max-reads should be available in both simplex and duplex basecalling. Can you share your full command again?
— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2147434816, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOOAM3Y22H4V6V6HH73ZFWYRJAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBXGQZTIOBRGY. You are receiving this because you were mentioned.Message ID: @.***>
Hi Richard,
I hope you are doing well.
Please, could you let us know if there are any update?
I am using "dorado-0.7.1-linux-x64/bin/dorado duplex" but I can't see the "--max-reads" option:
https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/duplex.cpp
Thanks! Cris
From: Maria Cristina Frias Lopez @.> Sent: Tuesday, June 4, 2024 3:22 PM To: nanoporetech/dorado @.>; nanoporetech/dorado @.> Cc: Mention @.>; Simon Heath @.>; Matthew John Ingham @.> Subject: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
Hi Richard,
I am using "dorado duplex", not "dorado basecaller" neither "dorado demux" (the links you send me).
Here you have the full command:
/home/cfrias/software/dorado-0.7.0-linux-x64/bin/dorado duplex
@.***
/scratch_isilon/groups/pbt/cfrias/ont02/<INPUT>
--min-qscore 10 --modified-bases 5mC_5hmC
--max-reads 10000
-vv > @._pod5pass.dorado-0.7.0-maxreads_10000.bam 2> @._pod5pass.dorado-0.7.0-maxreads_10000.log
I also installed the last version, but I can not see this parameter in the "duplex", just in the "simplex":
duplex
[cid:c514db7b-7515-4a82-9d86-a25dd680d7d1]
simplex
[cid:194c69a2-1bfa-462d-8aec-fc76017a6116]
Many thanks in advance,
Cris
From: Richard Harris @.> Sent: Tuesday, June 4, 2024 2:39 PM To: nanoporetech/dorado @.> Cc: Maria Cristina Frias Lopez @.>; Mention @.> Subject: Re: [nanoporetech/dorado] Dorado 0.5.3 - memory issues - BAM truncated? (Issue #768)
https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/basecaller.cpp#L473
https://github.com/nanoporetech/dorado/blob/release-v0.7/dorado/cli/demux.cpp#L106
--max-reads should be available in both simplex and duplex basecalling. Can you share your full command again?
— Reply to this email directly, view it on GitHubhttps://github.com/nanoporetech/dorado/issues/768#issuecomment-2147434816, or unsubscribehttps://github.com/notifications/unsubscribe-auth/BGMPNOOAM3Y22H4V6V6HH73ZFWYRJAVCNFSM6AAAAABGWOYSPCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBXGQZTIOBRGY. You are receiving this because you were mentioned.Message ID: @.***>
#855 has been resolved, which should fix this issue. For the --max-reads option in duplex please follow #1084.