HyperFastCgi icon indicating copy to clipboard operation
HyperFastCgi copied to clipboard

Probably race condition?

Open Evengard opened this issue 6 years ago • 1 comments

I've got a weird issue which I am trying to hunt down for a while. The issue appears only after some time running. After a while I see the process being in sort of hung state not accepting new connections and eventually just dying with CPU usage 100% on all cores... I tried to debug it with gdb with little to no success. The only thing I found that each of the stuck threads are stuck somewhere in "mono_conc_hashtable_lookup" (as per gdb line 174), seems like they are looping inside it. Trying to get backtrace of it (using instructions from http://www.mono-project.com/docs/debug+profile/debug/) lead me to this kind of backtrace:

#0  0x00000000006d3862 in mono_conc_hashtable_lookup (hash_table=0x2db2680, key=key@entry=0x7feae554fab0) at mono-conc-hashtable.c:174
174     in mono-conc-hashtable.c
#1  0x000000000042e76b in mono_jit_runtime_invoke (method="System.Collections.Generic.List`1:.cctor ()", obj=0x0, params=0x0, exc=0x7feb22498b58,
    error=0x7feb22498cd0) at mini-runtime.c:2646
2646    mini-runtime.c: No such file or directory.
#2  0x00000000005f88bb in do_runtime_invoke (method="System.Collections.Generic.List`1:.cctor ()", obj=<optimized out>, params=<optimized out>,
    exc=<optimized out>, error=0x7feb22498cd0) at object.c:2887
2887    object.c: No such file or directory.
#3  0x00000000005fc4c6 in mono_runtime_class_init_full (vtable=vtable("System.Collections.Generic.List`1"), error=error@entry=0x7feb22498cd0) at object.c:475
475     in object.c
#4  0x000000000042db65 in mono_jit_compile_method_with_opt (method="System.Collections.Generic.List`1:.ctor ()", opt=370239999, jit_only=<optimized out>,
    error=0x7feb22498cd0) at mini-runtime.c:2055
2055    mini-runtime.c: No such file or directory.
#5  0x00000000004ba36b in common_call_trampoline (regs=0x2db2680, regs@entry=0x7feb22498d88,
    code=0x479b8693 "H\213L$\020I\211O\020I\215G\020H\211L$\bH\213\320H\301\352\tH\201\342\377\377\177", m="System.Collections.Generic.List`1:.ctor ()", vt=0x0,
    vtable_slot=0x0, error=0x7feb22498cd0) at mini-trampolines.c:715
715     mini-trampolines.c: No such file or directory.
#6  0x00000000004bab4f in mono_magic_trampoline (regs=0x7feb22498d88,
    code=0x479b8693 "H\213L$\020I\211O\020I\215G\020H\211L$\bH\213\320H\301\352\tH\201\342\377\377\177", arg=0x7feae5b439a0, tramp=<optimized out>)
    at mini-trampolines.c:846
846     in mini-trampolines.c
#7  0x0000000040ac0387 in ?? ()
#8  0x00007feb22499a7d in ?? ()
#9  0xc0630e836f08e600 in ?? ()
#10 0x00007feb22498f50 in ?? ()
#11 0x00007feb22498d88 in ?? ()
#12 0x00007feaf9eb23d0 in ?? ()
#13 0x00007feb2bd36198 in ?? ()
#14 0x0000000000000000 in ?? ()

...which doesn't make a lot of sense other that we're stuck somewhere in some List constructor. Is it possible that we somehow got a race condition? If yes, on what List would that be?

Evengard avatar Mar 27 '18 07:03 Evengard

UPDATE: trying "Task" instead of "ThreadPool" lead me to the same error... Just quicker. As only one thread hung on this instead of multiple ones. It may not be a race condition, I guess... But what is it? Tbh I'm ready to debug it the most appropriate way, I just really don't know what to do as I'm not familiar with gdb (on the other hand kind of familiar with IDA, but I don't think I'll be able to reproduce it locally...)

Evengard avatar Mar 27 '18 07:03 Evengard