Segmentation fault at broken connection

[maniac0s]

Hello

My installation of Barnyard2 is supposed to connect over a VPN to a database. When this VPN does not reconnect fast enough, Barnyard2 crashes with segmentation fault.

database: mysql_error: Unknown MySQL server host 'some.ip" (110) database: Failed to logon to database 'snort' Segmentation fault

I am running Version 2.1.14 (Build 336) of Barnyard2

[maniac0s]

I again got a segfault. Can someone tell me what todo about this? A software shouldn't segfault on a broken connection, never ever. This is clearly a bug, even if the problem was caused by a connection problem, the software should be able to handle that and try to reconnect instead of messing up with the memory.

Aug 31 09:44:07 sen-ofc barnyard2[9449]: database: mysql_error: Can't connect to MySQL server on 'some.ip' (113) Aug 31 09:44:07 sen-ofc barnyard2[9449]: database: Failed to logon to database 'snort' Aug 31 09:44:07 sen-ofc kernel: barnyard2[9449]: segfault at 350 ip 00007f8181b2e4c0 sp 00007ffec9390048 error 4 in libmysqlclient.so.18.0.0[7f8181afd000+2b7000]

[binf]

A stacktrace would be helpfull. By2 is not "messing" the memory and it should reconnect without an issue. So posting a stacktrace or a coredump with your compiled binary would be the way to go. If you are sesitive about authentication information and such i would suggest that you post a coredump first. Also make sure you run against a by2 binary that has been build with debug symbols.

Make sure to do a make clean before re-running configure.

./configure --enable-debug

On Mon, Aug 31, 2015 at 3:53 AM, mindph [email protected] wrote:

I again got a segfault. Can someone tell me what todo about this? A software shouldn't segfault on a broken connection, never ever. This is clearly a bug, even if the problem was caused by a connection problem, the software should be able to handle that and try to reconnect instead of messing up with the memory.

Aug 31 09:44:07 sen-ofc barnyard2[9449]: database: mysql_error: Can't connect to MySQL server on 'some.ip' (113) Aug 31 09:44:07 sen-ofc barnyard2[9449]: database: Failed to logon to database 'snort' Aug 31 09:44:07 sen-ofc kernel: barnyard2[9449]: segfault at 350 ip 00007f8181b2e4c0 sp 00007ffec9390048 error 4 in libmysqlclient.so.18.0.0[7f8181afd000+2b7000]

— Reply to this email directly or view it on GitHub https://github.com/firnsy/barnyard2/issues/155#issuecomment-136292244.

[pauloangelo]

Hi @mindph,

We need more information to help you. By2 uses MySQL client lib. The segfault appears to be occurring there.

Try to recompile enabling debug, as posed by @binf and then use GDB to trace the problem, you can use the commands below. Remember that you MUST adjust the paths.

gdb /usr/local/by2/bin/barnyard .. [inside gdb] set follow-fork-mode child run -c /usr/local/by2/etc/barnyard2.conf -a /usr/local/snort/var/log/archive -f merged.log -d /usr/local/snort/var/log/ .. after the segfault .. bt

This should provide more information about the problem. You also can try to use "strace" or "ltrace" commands.

Did you upgrade MySQL lib after by2 compilation?! Is this problem happening after any system/lib upgrade? Maybe you need to recompile by2. You should also consider that segfaults can be associated with memory (hardware) problems. You can run a memory check to avoid this case.

[]'s

PA

[gades]

database: mysql_error: Can't connect to MySQL server on '127.0.0.1' (111) database: Failed to logon to database 'snorby'

Program received signal SIGSEGV, Segmentation fault. 0x000000369ac226f0 in mysql_thread_id () from /usr/lib64/mysql/libmysqlclient.so.16 Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.166.el6_7.3.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42.el6.x86_64 libcom_err-1.41.12-22.el6.x86_64 libpcap-1.4.0-4.20130826git2dbcaa1.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 mysql-libs-5.1.73-5.el6_6.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.x86_64 openssl-1.0.1e-42.el6.x86_64 tcl-8.5.7-6.el6.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x000000369ac226f0 in mysql_thread_id () from /usr/lib64/mysql/libmysqlclient.so.16 #1 0x00000000004276b5 in dbConnectionStatusMYSQL (pdbRH=0x7ffff7bb3970) at spo_database.c:4854 #2 0x0000000000427e1a in Insert (query=0x43f524 "BEGIN;", data=0x7ffff7b50010, inTransac=) at spo_database.c:3470 #3 0x0000000000428156 in BeginTransaction (data=0x7ffff7b50010) at spo_database.c:3246 #4 0x000000000042a1fe in Database (p=0x2a03410, event=0x5062200, event_type=7, arg=0x7ffff7b50010) at spo_database.c:2513 #5 0x0000000000415331 in CallOutputPlugins (out_type=, packet=0x2a03410, event=0x5062200, event_type=7) at plugbase.c:695 #6 0x000000000041627a in spoolerProcessRecord (spooler=0x5afc4a0, fire_output=1) at spooler.c:764 #7 0x0000000000416a1d in ProcessContinuous (dirpath=0x684a7c "/var/log/suricata", filebase=0x684e7c "unified2.alert", record_start=0, timestamp=1446074156) at spooler.c:590 #8 0x0000000000405a9b in Barnyard2Main (argc=11, argv=0x7fffffffdf68) at barnyard2.c:382 #9 0x000000369a01ed5d in __libc_start_main () from /lib64/libc.so.6 #10 0x00000000004032d9 in _start () (gdb)

[pauloangelo]

@gades

This problem occurred before (see URL below) and is occurring in mysql_thread_id .

https://groups.google.com/forum/#!msg/barnyard2-users/fDBb3NRn-fY/3op68LoEX1EJ

In MySQL documentation, below, the use of mysql_thread_id is not recommended. Perhaps is the case.

http://dev.mysql.com/doc/refman/5.0/en/mysql-thread-id.html

One alternative to test is changing the mysql_thread_id to a query "SELECT CONNECTION_ID()".

However, this code is a bit strange. Why is it requesting the thread id before a mysql_ping? What would this return in case of connection lost?

Try changing

// src/output-plugins/spo_database.c   line 4853
MYSQL_RetryConnection:
    /* mysql_ping() could reconnect and we wouldn't know */

    aThreadID = mysql_thread_id(pdbRH->dbdata->m_sock);

    ping_ret = mysql_ping(pdbRH->dbdata->m_sock);

    /* We might try to recover from this */

to

// src/output-plugins/spo_database.c   line 4853
MYSQL_RetryConnection:
    /* mysql_ping() could reconnect and we wouldn't know */

    aThreadID = pdbRH->pThreadID;
    //aThreadID = mysql_thread_id(pdbRH->dbdata->m_sock);

    ping_ret = mysql_ping(pdbRH->dbdata->m_sock);

    /* We might try to recover from this */

Recompile and try again.

Please, tell us if it solved or not.

Regards,

PA

[gades]

Yes, this return in case of connection lost and to check I do: service mysqld restart

(gdb) bt #0 0x000000369ac225a4 in mysql_ping () from /usr/lib64/mysql/libmysqlclient.so.16 #1 0x00000000004276b9 in dbConnectionStatusMYSQL (pdbRH=0x7ffff7bb3970) at spo_database.c:4857 #2 0x0000000000427e1a in Insert (query=0x43f524 "BEGIN;", data=0x7ffff7b50010, inTransac=) at spo_database.c:3470 #3 0x0000000000428156 in BeginTransaction (data=0x7ffff7b50010) at spo_database.c:3246 #4 0x000000000042a1fe in Database (p=0x2a03300, event=0x58afc70, event_type=7, arg=0x7ffff7b50010) at spo_database.c:2513 #5 0x0000000000415331 in CallOutputPlugins (out_type=, packet=0x2a03300, event=0x58afc70, event_type=7) at plugbase.c:695 #6 0x000000000041627a in spoolerProcessRecord (spooler=0x5afc390, fire_output=1) at spooler.c:764 #7 0x0000000000416a1d in ProcessContinuous (dirpath=0x684a7c "/var/log/suricata", filebase=0x684e7c "unified2.alert", record_start=0, timestamp=1446187882) at spooler.c:590 #8 0x0000000000405a9b in Barnyard2Main (argc=11, argv=0x7fffffffdf68) at barnyard2.c:382 #9 0x000000369a01ed5d in __libc_start_main () from /lib64/libc.so.6 #10 0x00000000004032d9 in _start () (gdb)