FastStack causing mysql2 disconnects?

[jdelStrother]

Hi there - I've been having problems using MiniProfiler's flamegraph functionality on a number of our pages. Queries would randomly fail, printing some variant of "Lost connection to MySQL server during query blah".

I can reproduce this with something simple like:

require 'mysql2'
require 'fast_stack'
client = Mysql2::Client.new(:host => "localhost", :username => "root", database:'foo')
FastStack.profile{client.query("SELECT * FROM users LIMIT 10000")}

resulting in :

Mysql2::Error: Lost connection to MySQL server during query
        from (irb):5:in `query'
        from (irb):5:in `block in irb_binding'
        from /Users/jon/.rvm/gems/ruby-2.0.0-p353@abweb/gems/fast_stack-0.1.0/lib/fast_stack.rb:22:in `profile_block'
        from /Users/jon/.rvm/gems/ruby-2.0.0-p353@abweb/gems/fast_stack-0.1.0/lib/fast_stack.rb:22:in `profile'
        from (irb):5
        from /Users/jon/.rvm/rubies/ruby-2.0.0-p353/bin/irb:12:in `<main>'

This is on ruby 2.0p353, mysql2 3.1.14, and mysql 5.5.25a on OS X 10.9. Anyone else run into anything similar or know any workarounds?

[SamSaffron]

What happens if you bump up the fidelity, only collect a sample every 2-4ms.

On Mon, Dec 2, 2013 at 4:23 AM, Jonathan del Strother < [email protected]> wrote:

Hi there - I've been having problems using MiniProfiler's flamegraph functionality on a number of our pages. Queries would randomly fail, printing some variant of "Lost connection to MySQL server during query blah".

I can reproduce this with something simple like:

require 'mysql2'require 'fast_stack'client = Mysql2::Client.new(:host => "localhost", :username => "root", database:'foo')FastStack.profile{client.query("SELECT * FROM users LIMIT 10000")}

resulting in :

Mysql2::Error: Lost connection to MySQL server during query from (irb):5:in query' from (irb):5:inblock in irb_binding' from /Users/jon/.rvm/gems/ruby-2.0.0-p353@abweb/gems/fast_stack-0.1.0/lib/fast_stack.rb:22:in profile_block' from /Users/jon/.rvm/gems/ruby-2.0.0-p353@abweb/gems/fast_stack-0.1.0/lib/fast_stack.rb:22:inprofile' from (irb):5 from /Users/jon/.rvm/rubies/ruby-2.0.0-p353/bin/irb:12:in `

'

This is on ruby 2.0p353, mysql2 3.1.14, and mysql 5.5.25a on OS X 10.9. Anyone else run into anything similar or know any workarounds?

— Reply to this email directly or view it on GitHubhttps://github.com/SamSaffron/fast_stack/issues/2 .

[jdelStrother]

It errors out less frequently, but I needed to bump it to 5ms to be able to successfully complete a request (even then, it maybe works half the time).

[SamSaffron]

Any chance you can create a standalone sample, I would like to have a play with it.

On Mon, Dec 2, 2013 at 9:42 AM, Jonathan del Strother < [email protected]> wrote:

It errors out less frequently, but I needed to bump it to 5ms to be able to successfully complete a request (even then, it maybe works half the time).

— Reply to this email directly or view it on GitHubhttps://github.com/SamSaffron/fast_stack/issues/2#issuecomment-29585534 .

[jdelStrother]

How's this? https://gist.github.com/jdelStrother/7747010 It's selecting data from information_schema.COLUMNS, just for ease of being able to run it regardless of your db schema, but I see similar behaviour in my regular "SELECT * FROM users" style queries.

It's somehow related to the quantity of data returned from the query, rather than the amount of time taken to perform the query. Not sure if mysql2 is switching to a different mode when it's fetching a lot of data, or if it's maybe related to the amount of data that can fit into a single packet or something..? Wonder if @brianmario might be able to shed some light on the problem.

[SamSaffron]

I am unable to repro this, tried on a linode and on local ... @tmm1 have you ever seen anything like this?

What version of the mysql2 gem are you on? What OS? What architecture (x64 / x32 ) ?

[jdelStrother]

Curious. So far I've been on x86 ruby 2.0p353, mysql2 3.1.14, and mysql 5.5.25a on OS X 10.9.0, on a Macbook Air. I get the same problem with ruby 1.9.3-p194.

On my Mac Mini (same OS & ruby, but mysql 5.5.28), I need to bump the LIMIT in that gist to 134 before it shows the problem.

I had a much harder time reproducing it on our SmartOS servers running 32 bit ruby 2.0.0p353, with mysql 5.5.21. I had to increase the LIMIT value and select from some of our larger tables, and even then couldn't seem to produce it 100% of the time. Not quite sure what else to try...

[tmm1]

I have seen this before, but I'm not sure what the exact circumstances required are. IIRC something inside libmysqlclient doesn't like getting interrupted. I've been using a 1ms cpu itimer and it hasn't happened since. We also upgraded our libmysqlclient recently.

[tmm1]

From http://bugs.mysql.com/bug.php?id=65956

Also, this behavior was changed in MySQL 5.6 and up. For the client library, interrupted I/O operations are now always retried.

[BMorearty]

I've been hitting this problem as well when doing flamegraphs this week. Using mysql2 0.3.15, Mac OSX 10.9.2 64-bit. To @tmm1's point about MySQL versions, I'm on MySQL 5.5.29.

The workaround of increasing the sample rate has worked for me. (1 or 2ms typically does the trick.)

[jdelStrother]

FWIW, I still get this in OS X 10.10.1, with ruby 2.1.4, mysql2 0.3.17, and mysql 5.6.21

[SamSaffron]

seen this before, a reasonable workaround is just to use stackprof for this in Ruby 2.1 which should not suffer from the issue (in theory) - latest flamegraph supports it just require the gem

On Thu, Dec 4, 2014 at 2:48 AM, Jonathan del Strother < [email protected]> wrote:

FWIW, I still get this in 10.10.1, with ruby 2.1.14, mysql2 0.3.17, and mysql 5.6.21

— Reply to this email directly or view it on GitHub https://github.com/SamSaffron/fast_stack/issues/2#issuecomment-65431120.

[jdelStrother]

Thanks for the tip, that works great