pinba_engine
pinba_engine copied to clipboard
tony2001
crash in ha_pinba:close()
May 29 08:03:51 perfccc mysqld: 120529 8:03:51 - mysqld got signal 11 ; May 29 08:03:51 perfccc mysqld: This could be because you hit a bug. It is also possible that this binary May 29 08:03:51 perfccc mysqld: or one of the libraries it was linked against is corrupt, improperly built, May 29 08:03:51 perfccc mysqld: or misconfigured. This error can also be caused by malfunctioning hardware. May 29 08:03:51 perfccc mysqld: We will try our best to scrape up some info that will hopefully help diagnose May 29 08:03:51 perfccc mysqld: the problem, but since we have already crashed, something is definitely wrong May 29 08:03:51 perfccc mysqld: and this may fail. May 29 08:03:51 perfccc mysqld: May 29 08:03:51 perfccc mysqld: key_buffer_size=268435456 May 29 08:03:51 perfccc mysqld: read_buffer_size=1048576 May 29 08:03:51 perfccc mysqld: max_used_connections=97 May 29 08:03:51 perfccc mysqld: max_threads=200 May 29 08:03:51 perfccc mysqld: threads_connected=37 May 29 08:03:51 perfccc mysqld: It is possible that mysqld could use up to May 29 08:03:51 perfccc mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)max_threads = 2107414 K May 29 08:03:51 perfccc mysqld: bytes of memory May 29 08:03:51 perfccc mysqld: Hope that's ok; if not, decrease some variables in the equation. May 29 08:03:51 perfccc mysqld: May 29 08:03:51 perfccc mysqld: Thread pointer: 0x7fd3f812d0a0 May 29 08:03:51 perfccc mysqld: Attempting backtrace. You can use the following information to find out May 29 08:03:51 perfccc mysqld: where mysqld died. If you see no messages after this, something went May 29 08:03:51 perfccc mysqld: terribly wrong... May 29 08:03:51 perfccc mysqld: stack_bottom = 0x7fd4073afe88 thread_stack 0x40000 May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x29) [0x7fd45ac945d9] May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(handle_segfault+0x435) [0x7fd45a9ef105] May 29 08:03:51 perfccc mysqld: /lib/libpthread.so.0(+0xeff0) [0x7fd45a26aff0] May 29 08:03:51 perfccc mysqld: /usr/lib/mysql/plugin/libpinba_engine.so(ha_pinba::close()+0xab) [0x7fd43fc9ffab] May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(closefrm(st_table, bool)+0x10d) [0x7fd45aa4319d] May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(+0x39dbd9) [0x7fd45aa35bd9] May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(my_hash_delete+0x334) [0x7fd45ac8a1b4] May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(close_thread_tables(THD_)+0x244) [0x7fd45aa356f4] May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(dispatch_command(enum_server_command, THD_, char_, unsigned int)+0x1cc) [0x7fd45aa0125c] May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(do_command(THD_)+0xea) [0x7fd45aa0293a] May 29 08:03:51 perfccc mysqld: /usr/sbin/mysqld(handle_one_connection+0x22d) [0x7fd45a9f59dd] May 29 08:03:51 perfccc mysqld: /lib/libpthread.so.0(+0x68ca) [0x7fd45a2628ca] May 29 08:03:51 perfccc mysqld: /lib/libc.so.6(clone+0x6d) [0x7fd458daa86d] May 29 08:03:51 perfccc mysqld: May 29 08:03:51 perfccc mysqld: Trying to get some variables. May 29 08:03:51 perfccc mysqld: Some pointers may be invalid and cause the dump to abort. May 29 08:03:51 perfccc mysqld: Query (0x7fd3f8022430): is an invalid pointer May 29 08:03:51 perfccc mysqld: Connection ID (thread ID): 742694 May 29 08:03:51 perfccc mysqld: Status: NOT_KILLED May 29 08:03:51 perfccc mysqld: May 29 08:03:51 perfccc mysqld: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains May 29 08:03:51 perfccc mysqld: information that should help you find out what is causing the crash.
Versions: mysql Ver 14.14 Distrib 5.1.58, for debian-linux-gnu (x86_64) using readline 6.1
mysql-server: Installed: 5.1.58-1~dotdeb.1
pinba-mysql-engine: Installed: 0.0.6-1.dotdeb.4
php5-pinba: Installed: 5.3.10-1~dotdeb.1
looks like share(0) missed in ha_pinba constructor, and if(!share)free_share(share) should be in ::close(). I'll test that tomorrow.
? Didn't get your comment. How to reproduce this?
Please try Git version first and make sure you use the correct (and correctly configured) MySQL headers when building Pinba.
I can't reproduce it stable -- i just have system in production, and script that every 5 minutes doing SELECT from aggregate tables, and TRUNCATE after. Already twice mysql server was crashed, both times -- crash in ha_pinba::close()
TRUNCATE? Why would you need that? Please try fresh Git checkout, I might have fixed that already.
I need to get exact values for 5min -- so i need read & clear, so averages will be correct
But they are the exact values for the last 5 mins, every moment you look at them if stats_history = 300 sec.
thanks, i'll try that, but that doesn't fix possibility of mysql server crash, caused by freeing non-initialized share pointer
Of course it doesn't, but I still don't get what you're talking about. What pointer and why do you think it might be uninitialized?
that line are caused crash, but note that free_share are inlined, so crash in free_share() https://github.com/tony2001/pinba_engine/blob/master/src/ha_pinba.cc#L1650
here is constructor: https://github.com/tony2001/pinba_engine/blob/master/src/ha_pinba.cc#L1616 and it doesn't nullify share pointer
The share variable is initialized using get_share() call, which either returns a valid pointer or NULL and is called in open() method. And I have some serious doubts that MySQL doesn't call open() before calling close().
Yes, it can return NULL.... But free_share doesn't check is incoming pointer null.
Ok, if you're sure that's the problem, please add if (share) { } around free_share() call and check if it helps. I couldn't find any such checks in other storage engines, so I assumed that MySQL is smart enough not to call close() before open() or if open() returned an error.