MariaDB 10.0 Client (Ubuntu 16.04) Crashes Bedrock

[willfong]

Bedrock version: 43234709c216b19ae7ce7ea7a5219467bf50e773

Using the MariaDB 10.0 client on Ubuntu 16.04LTS seems to crash bedrock.

Test case:

root@ubuntu6:~# service bedrock start
root@ubuntu6:~# ps aux|grep bedrock
root      5531  0.0  0.3 400332  7912 ?        Ssl  04:34   0:00 /usr/sbin/bedrock -fork -nodeName bedrock -db /var/lib/bedrock/bedrock.db -serverHost 0.0.0.0:8888 -nodeHost 0.0.0.0:8889 -priority 200 -pidfile /var/run/bedrock.pid -quorumCheckpoint 100 -readThreads 4 -plugins status,db,jobs,cache,mysql -v -cache 10001
root      5538  0.0  0.0  14224  1020 pts/0    S+   04:34   0:00 grep --color=auto bedrock
root@ubuntu6:~# bedrock -version
43234709c216b19ae7ce7ea7a5219467bf50e773
root@ubuntu6:~# mysql --version
mysql  Ver 15.1 Distrib 10.0.28-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
root@ubuntu6:~# mysql -h127.0.0.1 -e "SELECT 1 AS foo, 2 AS bar;"
ERROR 2006 (HY000) at line 1: MySQL server has gone away
root@ubuntu6:~#

Unfortunately, there doesn't seem to be anything logged:

root@ubuntu6:~# cat /var/log/syslog
Dec 23 04:42:14 ubuntu6 bedrock: xxxxx (STCPServer.cpp:57) operator() [main] [dbug] Accepting socket from '127.0.0.1:58732' on port 'localhost:3306'
Dec 23 04:42:14 ubuntu6 bedrock: xxxxx (BedrockServer.cpp:473) postSelect [main] [info] Plugin 'MySQL' accepted a socket from '127.0.0.1:58732'
Dec 23 04:42:14 ubuntu6 bedrock: xxxxx (MySQL.cpp:252) onPortAccept [main] [info] {MySQL} Accepted MySQL request from '127.0.0.1:58732'
Dec 23 04:42:14 ubuntu6 bedrock: xxxxx (MySQL.cpp:263) onPortRecv [main] [dbug] {MySQL} Received command #-123: '0B00000185A0000000726F6F740000'
Dec 23 04:42:14 ubuntu6 bedrock: xxxxx (MySQL.cpp:338) onPortRecv [main] [info] {MySQL} Sending OK
Dec 23 04:42:14 ubuntu6 bedrock: xxxxx (MySQL.cpp:263) onPortRecv [main] [dbug] {MySQL} Received command #3: '210000000373656C65637420404076657273696F6E5F636F6D6D656E74206C696D69742031'
Dec 23 04:42:14 ubuntu6 bedrock: xxxxx (MySQL.cpp:269) onPortRecv [main] [info] {MySQL} Processing query 'select @@version_comment limit 1'
root@ubuntu6:~# ps aux|grep bedrock
root      5735  0.0  0.0  14224  1004 pts/0    S+   04:42   0:00 grep --color=auto bedrock
root@ubuntu6:~# 

[quinthar]

@willfong Can you clarify that the following steps will reproduce it:

1. Install Ubuntu 16.04
2. Run sudo apt-get install mariadb
3. Install Bedrock per the instructions at http://bedrockdb.com
4. Run service bedrock start
5. Run mysql -h127.0.0.1 -e "SELECT 1 AS foo, 2 AS bar;"

Is there anything more to it than that -- no special commands or anything?

Tips for fixing:

Here are some pointers to anyone who wants to fix this:

• The MySQL compatibility layer is entirely in MySQL.cpp/h
• The getPort() command is what causes the plugin to open a port
• Each time a connection is received on that port, onPortAccept() is called
• The MySQL protocol is a bit unusual in that the server is the first to announce itself on a new connection, which it does to here
• All MySQL packets are parsed and constructed using the simple MySQLPacket struct
• Specifically, it sends back the results of MySQLPacket::serializeHandshake()
• It's possible, but unlikely that that's where it crashes. Odds are the client receives that handshake successfully, and then responds with a request that crashes Bedrock.
• That request would be processed in onPortRecv(), which is called whenever a socket accepted on the plugin's port receives data
• To see the most verbose logging, start bedrock using the -v flag. This will cause its SDEBUG() log lines to execute (eg, like here)
• MySQL is a pretty harsh and unforgiving binary protocol, so this often requires manually inspecting the hex-encoded request and comparing against the rather confusion (to me, at least) binary protocol documentation.
• In particular, it uses a super-optimized length-encoded integer format, which you can see how to encode here.
• Incidentally, I'd suggest adding a SINFO("Executing true query") line here, as that's the only code path I see that doesn't log what's happening clearly.
• However, it looks from the log provided that it's crashing on the query select @@version_comment limit 1, which should be processed here, and should in theory successfully return this value here.
• But that's a nasty little bit of code, so I could easily imagine a crash bug existing in there.

Regardless, if this is a reliably reproducible problem, that happens during connection setup, it should be relatively easy to fix -- especially since it seems to be happening after the packet itself was parsed. (Bugs during parsing are a lot harder to fix.)

Any volunteers to take it on?

[quinthar]

FYI, posted an Upwork job here if anybody would like to earn a quick $500: https://www.upwork.com/jobs/_~013f98c7e3d216f1a1

[willfong]

@quinthar Yes, those steps are correct. Thanks for looking into this!

[Vadius]

@willfong, could you check if the bug reproduced on 7ece319 ? I've tried to do it, but all work fine:

// tty.1
$ ./bedrock -clean -v -plugins status,db,jobs,cache,mysql -cache 10001

// tty.2
$ mysql --version
mysql  Ver 15.1 Distrib 10.0.28-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
$ mysql -h127.0.0.1 -e "SELECT 1 AS foo, 2 AS bar;"
+------+------+
| foo  | bar  |
+------+------+
|    1 |    2 |
+------+------+

[quinthar]

@willfong - Can you please rebuild from the master branch and test to see if you can reproduce this issue? Thanks!

[quinthar]

@willfong - Eager for your comments here, btw.

[willfong]

@quinthar Hi, Sorry about the delay. This week is a bit busy for me, but I can check it out next week. Thank you very much for your fast resolution to this 😄

[willfong]

Sorry, this week has been terribly busy for me. Can we mark this as resolved for now? When I get some time again, I can open a new issue if I encounter this again. Thank you again for the prompt resolution!

[Vadius]

@willfong , I can send you already built Bedrock binaries in tgz, so you should just run your test. If it's ok, send me a mail to vadius[]vadius.ru, I'll reply with the archive.

[ghost]

@willfong should be fixed by https://github.com/Expensify/Bedrock/pull/127