Rails parallel-testing raises Bad file descriptor (Errno::EBADF) when using mysql2 installed with MySQL 8.0

[johnmaxwell]

In brief

We encounter a concurrency issue when using mysql2 0.5.3 that is installed using MySQL 8.0.23. If you install the gem using MySQL 5.7 or MariaDB 10.5.8 the issue doesn't present.

Description of problem

I apologize in advance that I can't be more specific than reporting this issue in the context of a Rails concurrency feature, but please bear with me since there is a reliable use-case for reproduction.

We use Homebrew to install MySQL 8.0 on Macs we use for development, and when installing the mysql2 0.5.3 gem, we point it to this installed MySQL: gem install mysql2 --with-mysql-dir="/usr/local/opt/mysql" Rails has a feature called parallel-testing where it uses DRb to run a test suite across several processes.

In this configuration, when we trying to run our Rails test-suite in parallel, we are stymied by the following Errno::EBADF errors that are reliably emitted during a test run.

$ PARALLEL_WORKERS=2 ./bin/rails test
Running via Spring preloader in process 9204
Run options: --seed 6137

# Running:

......#<Thread:0x00007ffcb81416c8 /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1259 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	10: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1262:in `block in make_pool'
	 9: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1237:in `_execute'
	 8: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
	 7: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
	 6: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1239:in `block in _execute'
	 5: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1268:in `block (2 levels) in make_pool'
	 4: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1268:in `each'
	 3: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1270:in `block (3 levels) in make_pool'
	 2: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1334:in `alive?'
	 1: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1002:in `alive?'
/Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1002:in `wait_readable': Bad file descriptor (Errno::EBADF)
#<Thread:0x00007ffcb81416c8 /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1259 run> terminated with exception (report_on_exception is true):
Traceback (most recent call last):
	10: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1262:in `block in make_pool'
	 9: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1237:in `_execute'
	 8: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/monitor.rb:202:in `mon_synchronize'
	 7: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/monitor.rb:202:in `synchronize'
	 6: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1239:in `block in _execute'
	 5: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1268:in `block (2 levels) in make_pool'
	 4: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1268:in `each'
	 3: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1270:in `block (3 levels) in make_pool'
	 2: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1334:in `alive?'
	 1: from /Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1002:in `alive?'
/Users/john/.rbenv/versions/2.7.2/lib/ruby/2.7.0/drb/drb.rb:1002:in `wait_readable': Bad file descriptor (Errno::EBADF)

If you bypass the the Rails parallel testing feature by running the tests serially, the problem does not present, so this is a problem that only occurs when doing the kind of concurrency that Rails is doing in a parallel-test run. The PARALLEL_WORKERS environment variable is how you can control whether Rails runs the tests in parallel or serially. To run the test suite serially and see it run to completion successfully, specify PARALLEL_WORKERS=1:

$ PARALLEL_WORKERS=1 ./bin/rails test
Running via Spring preloader in process 77228
Run options: --seed 8561

# Running:

........................................................................................................................................................................................................

Finished in 16.759488s, 11.9335 runs/s, 11.9335 assertions/s.
200 runs, 200 assertions, 0 failures, 0 errors, 0 skips

We have determined that the success/failure of this feature depends on which version of MySQL we install the mysql2 gem with. Installing the gem with MySQL 5.7 or MariaDB 10.5.8 allows the tests to run successfully in parallel.

The reproduction case is a boilerplate Rails application with minimal customization other than to configure it to use MySQL, add a simple ActiveRecord model, and a test file with 200 test cases that provide a basis for reproduction. Please see: https://github.com/johnmaxwell/parallelmysql61

Reproduction steps

1. Pull down https://github.com/johnmaxwell/parallelmysql61
2. brew install mysql (Or install MySQL 8.0 using the Oracle packaged installer.)
3. brew services start mysql
4. gem install mysql2 --with-mysql-dir="/usr/local/opt/mysql" (Or point it to the Oracle packaged install directory.)
5. bundle install
6. Configure database.yml as needed to connect to your mysql. If you have a root user with no password (an unfortunate default) it may work as-is.
7. ./bin/rails db:create
8. PARALLEL_WORKERS=2 ./bin/rails test

You should see it fail as described above. To see it work successfully, uninstall you mysql2 gem and re-install it with the MySQL 5.7 client library:

1. gem uninstall mysql2
2. brew install [email protected]
3. gem install mysql2 -- --with-mysql-dir="/usr/local/opt/[email protected]/"
4. ./bin/spring stop to make sure the Spring preloader doesn't confound us
5. PARALLEL_WORKERS=2 ./bin/rails test

The tests run should run to completion with no errors.

Here's the original issue that I opened with Rails about this issue: https://github.com/rails/rails/issues/41176

System configuration

mysql2 version: 0.5.3 MySQL version: 8.0.23 Ruby version: 2.7.2p137 Computer: Intel Mac running MacOS 10.15.7 Rails version: 6.0.3.4, 6.1.1 & rails/rails:e8d7181ad6fd040b60c25752c1a60d939e4a8505

[coderdave]

I was having this issue too, thanks for the help.

You can also change the number of workers in test_helper.rb so you don't have to remember to type it in every time you want to run the test suite.

...

class ActiveSupport::TestCase
  # Don't run tests in parallel since it causes errors
  parallelize(workers: 1)

  # Setup all fixtures in test/fixtures/*.yml for all tests in alphabetical order.
  fixtures :all
end

[fzhunt]

Besides compiling mysql2 with the mysql 5.7 client as is mentioned in https://github.com/rails/rails/issues/41176, have any other solutions to this issue been found? We'd prefer to be able to use the default compiled version of mysql2 just for future compatibility's sake, but parallel testing has been incredibly useful to us.