Segmentation fault with libmysql 5.7.28

[stissot]

Since our Ubuntu 18.04 servers were upgraded to libmysql 5.7.28 (security), we have a segmentation fault using mysql2 0.5.2 to connect to MariaDB 10.3/10.2 databases. If we downgrade the lib to 5.7.27, no more error.

  * [create MySQL database and user for zabbix] action create/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2/mysql2.so: [BUG] Segmentation fault at 0x0000000000000000
ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0050 p:-7355282 s:0256 e:000255 TOP    [FINISH]
c:0049 p:---- s:0253 e:000252 CFUNC  :require
c:0048 p:0110 s:0248 e:000247 METHOD /opt/chef/embedded/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59
c:0047 p:0245 s:0236 e:000235 TOP    /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/mysql2-0.5.2/lib/mysql2.rb:33 [FINISH]
c:0046 p:---- s:0232 e:000231 CFUNC  :require
c:0045 p:0091 s:0227 e:000226 RESCUE /opt/chef/embedded/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:135
c:0044 p:0376 s:0223 e:000222 METHOD /opt/chef/embedded/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:39

Do you need more details to investigate?

[wdeviers]

What version of Chef are you using?

[stissot]

What version of Chef are you using?

Chef client 14.5.33

[wdeviers]

Okay. I'm on 14.11. I think what's happening is that gems compiled inside Chef link against the system libssl but run against libssl in /opt/chef/embedded/lib. I've been trying to figure out a way around it for a few hours now. I thought I'd fixed it by calling ShellOut to compile the gem manually instead of using chef_gem, but it doesn't work all the time. Just sometimes.

[wdeviers]

So it appears that if I use libperconaserverclient18-dev as well as:

gem_install = Mixlib::ShellOut.new('/opt/chef/embedded/bin/gem install --http-proxy http://proxy.fake:3128 mysql2 -- --with-ldflags="-L/opt/chef/embedded/lib/ -R/opt/chef/embedded/lib/" --with-cppflags=-I/opt/chef/embedded/include/ ').run_command it works. Maybe. I haven't pushed it past my local dev but this seems to solve the issues.

[pantras]

We're no more seeing that crash now that we're doing an apt-get update -y before installing libmysqlclient-dev - and mysql2 gem. HTH

(I rewrote the commit mentionned above in a slightly cleaner way: https://github.com/pantras/libmysqlclient-cookbook/commit/f3e8f0b188604470b1ece279e76358cbc4e767dc)

[acoulton]

We have the same issue - unfortunately the workaround with the gem install flags isn't working for us, nor is doing an apt-get update. This is with both 0.5.2 and the just-released 0.5.3

[acoulton]

OK - I wasn't keen to downgrade the system libmysqlclient packages so I manually followed @pantras steps to get a local copy of the older libs and includes for the 5.7.27 version. Then I rolled those up into a tar, and wrote a recipe to provision those into the /opt/chef/embedded libs and includes directories. Then used @wdeviers command to specify the embedded libs and includes directories when installing the gem. See https://github.com/ingenerator/chef-ingenerator-mysql/commit/e4d705dde5a8eea595d11adba77be855077dbacd for the full thing.

This seems to be working, and means that although chef is using the older client the rest of the system is still on 5.7.28 and able to be bumped up again in future.

We're in the process of moving off chef anyway so hopefully this will be enough of a sticking plaster for the short term.

[sentience]

@acoulton Thank you for sharing that! Your work-around is working for me as well, except that I needed to change the syntax on the --with-ldflags from -R (which my gcc was saying it didn't recognize) to -Wl,-R,:

execute 'install and compile chef mysql2 gem' do
  action :nothing
  command '/opt/chef/embedded/bin/gem install mysql2 -v 0.5.2 -- --with-ldflags="-L/opt/chef/embedded/lib/ -Wl,-R,/opt/chef/embedded/lib/" --with-cppflags=-I/opt/chef/embedded/include/'
end

@stissot A more permanent solution for this would be appreciated. If you decide it's not practically fixable, we may have to move off Chef as well.

[zavan]

I'm having the same problem with 5.7.29. The flags and apt update didn't do it for me, I had to resort to manually installing 5.7.27. This is how I did it in a more "idiomatic" way:

package 'zlib1g-dev' # This is a dependency of the libs bellow

remote_file '/tmp/mysql-common_5.8+1.0.4_all.deb' do
  source 'https://launchpadlibrarian.net/353280751/mysql-common_5.8+1.0.4_all.deb'
  show_progress true
end

remote_file '/tmp/libmysqlclient20_5.7.27-0ubuntu0.18.04.1_amd64.deb' do
  source 'https://launchpadlibrarian.net/434319442/libmysqlclient20_5.7.27-0ubuntu0.18.04.1_amd64.deb'
  show_progress true
end

remote_file '/tmp/libmysqlclient-dev_5.7.27-0ubuntu0.18.04.1_amd64.deb' do
  source 'https://launchpadlibrarian.net/434319424/libmysqlclient-dev_5.7.27-0ubuntu0.18.04.1_amd64.deb'
  show_progress true
end

dpkg_package 'mysql-common' do
  source '/tmp/mysql-common_5.8+1.0.4_all.deb'
end

dpkg_package 'libmysqlclient20' do
  source '/tmp/libmysqlclient20_5.7.27-0ubuntu0.18.04.1_amd64.deb'
end

dpkg_package 'libmysqlclient-dev' do
  source '/tmp/libmysqlclient-dev_5.7.27-0ubuntu0.18.04.1_amd64.deb'
end

# Installs the mysql2 gem in chef's ruby
chef_gem 'mysql2' do
  version '0.5.3'
end

[DocX]

We have same issue with libmysqlclient 5.7.30 from ubuntu 18.10.

Rolling back to 5.7.27 is not option for us. But swapping to libmariadbclient-dev instead of libmysqlclient-dev solved the problem for us.

[mromero-convoso]

Looking for a clean solution for this one. I'm using the MariaDB cookbook to install MariaDB, and 'libmariadbclient-dev' is replaced by 'libmariadb-dev'. I'm running Ubuntu 20.04 LTS, and experiencing the segfault anytime I require 'mysql2'