kalloc panic

[armintoepfer]

Hi Heng,

our IsoSeq clustering algo relies on mm2 and so far no problems in the last 1.5 years. We just hit a reproducible kalloc panic:

[kfree] The end of a free block enters the allocated block.

The problem with reproducibility is that this has only occurred so far with >4M CCS input reads and thus can't be quickly reproduced. Do you have any idea what is going on? I'll set a breakpoint in kalloc.c:123 (or rather l108 for my mm2 version 2.15) and wait for the debug mode to finish.

Armin

[lh3]

This is typically caused by heap corruption – i.e. there is probably a memory violation elsewhere. You may compile minimap2 with asan and run it with option --no-kalloc. Asan might be able to capture the real violation.

[armintoepfer]

Since we are using mm2 as a library, reproducing outside is tricky because this step uses ~300k sequences as references, I'm still working on reproducing it with the minimap2 binary.

So far, I was able to run IsoSeq in debug mode with asan and deactivated kalloc. I was able to create a core file upon asan failure.

#0  0x00002b89594b01f7 in raise () from /lib64/libc.so.6
#1  0x00002b89594b18e8 in abort () from /lib64/libc.so.6
#2  0x00002b895530ce3a in __sanitizer::Abort () at /build/gcc-tc-passfinal-targ2targ//src/gcc/libsanitizer/sanitizer_common/sanitizer_posix_libcdep.cc:145
#3  0x00002b8955312df4 in __sanitizer::Die () at /build/gcc-tc-passfinal-targ2targ//src/gcc/libsanitizer/sanitizer_common/sanitizer_termination.cc:57
#4  0x00002b8955300f40 in __asan::ScopedInErrorReport::~ScopedInErrorReport (this=0x2b89657d685e, __in_chrg=<optimized out>) at /build/gcc-tc-passfinal-targ2targ//src/gcc/libsanitizer/asan/asan_report.cc:181
#5  0x00002b8955300aca in __asan::ReportGenericError (pc=47868837381982, bp=bp@entry=47869113234624, sp=sp@entry=47869113234616, addr=105692442322460, is_write=is_write@entry=false, access_size=access_size@entry=4, exp=0, fatal=true) at /build/gcc-tc-passfinal-targ2targ//src/gcc/libsanitizer/asan/asan_errors.h:333
#6  0x00002b895530127d in __asan::__asan_report_load4 (addr=<optimized out>) at /build/gcc-tc-passfinal-targ2targ//src/gcc/libsanitizer/asan/asan_rtl.cc:111
#7  0x00002b89550c475e in mm_sync_regs (km=0x0, n_regs=2, regs=0x61c02f085880) at ../subprojects/minimap2/hit.c:230
#8  0x00002b89550c6d0c in mm_join_long (km=0x0, opt=0x7fff3b2f3ac0, qlen=6437, n_regs_=0x2b89657d76f0, regs=0x61c02f085880, a=0x62e001830400) at ../subprojects/minimap2/hit.c:352
#9  0x00002b89550db8bc in chain_post (opt=0x7fff3b2f3ac0, max_chain_gap_ref=500, mi=0x60601a0a43a0, km=0x0, qlen=6437, n_segs=1, qlens=0x2b89657d7980, n_regs=0x2b89657d76f0, regs=0x61c02f085880, a=0x62e001830400) at ../subprojects/minimap2/map.c:255
#10 0x00002b89550dda3e in mm_map_frag (mi=0x60601a0a43a0, n_segs=1, qlens=0x2b89657d7980, seqs=0x2b89657d7a00, n_regs=0x2b89657d7ae0, regs=0x2b89657d79c0, b=0x602070791ab0, opt=0x7fff3b2f3ac0, qname=0x604014f6be90 "m64012_190830_234844/178784645/ccs") at ../subprojects/minimap2/map.c:351
#11 0x00002b89550debc1 in mm_map (mi=0x60601a0a43a0, qlen=6437, seq=0x623031c37500 "GGTTGGCCGCGCGCGGTAAGGGCGGGCGCCCGGAGAGGCAGGAGCGACTGCGGAGTGCTGGTGCGCCGCGCCCTGGTCTGCATCGGGCGCGGGCTGGCTCAAGCCGGAGTCCCGAGCAAGTGCGGGGGCTGCGCGTCCAGGGGAGCTGCTGTTGCCCCAGGAGACGACTGTGGAGCTGAGCTGTGGAGTGGGGCCACTGC"..., n_regs=0x2b89657d7ae0, b=0x602070791ab0,
    opt=0x7fff3b2f3ac0, qname=0x604014f6be90 "m64012_190830_234844/178784645/ccs") at ../subprojects/minimap2/map.c:392

(gdb) f 7
#7  0x00002b89550c475e in mm_sync_regs (km=0x0, n_regs=2, regs=0x61c02f085880) at ../subprojects/minimap2/hit.c:230
230			else if (r->parent >= 0 && tmp[r->parent] >= 0)
(gdb) l 226
221		tmp = (int*)kmalloc(km, n_tmp * sizeof(int));
222		for (i = 0; i < n_tmp; ++i) tmp[i] = -1;
223		for (i = 0; i < n_regs; ++i)
224			if (regs[i].id >= 0) tmp[regs[i].id] = i;
225		for (i = 0; i < n_regs; ++i) {
226			mm_reg1_t *r = &regs[i];
227			r->id = i;
228			if (r->parent == MM_PARENT_TMP_PRI)
229				r->parent = i;
230			else if (r->parent >= 0 && tmp[r->parent] >= 0)
(gdb) p r->parent
$1 = 3
(gdb) p n_tmp
$2 = 3

Of course this will throw. Would you have time if I pass you the core file and the debug binary?