sqlite3-ruby icon indicating copy to clipboard operation
sqlite3-ruby copied to clipboard

Published 20 hours ago • verified publisher icon sparklemotion

Bundler 2.3.23 on Debian Bullseye isn't installing the native platform gem

Open oliverklee opened this issue 2 years ago • 11 comments

Switching from version 1.5.2 to 1.5.3, installing this gem via bundle install on our local Docker-based x86_64 dev environment fails:

Installing sqlite3 1.5.3 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /usr/local/bundle/gems/sqlite3-1.5.3/ext/sqlite3
/usr/local/bin/ruby -I /usr/local/lib/ruby/site_ruby/3.1.0 extconf.rb
Building sqlite3-ruby using packaged sqlite3.
Extracting sqlite-autoconf-3390400.tar.gz into tmp/x86_64-linux-gnu/ports/sqlite3/3.39.4... OK
Running 'configure' for sqlite3 3.39.4... OK
Running 'compile' for sqlite3 3.39.4... OK
Running 'install' for sqlite3 3.39.4... OK
Activating sqlite3 3.39.4 (from /usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4)...

Could not configure the build properly (pkg_config). Please install either the `pkg-config` utility or the `pkg-config` rubygem.

*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
        --with-opt-dir
        --without-opt-dir
        --with-opt-include
        --without-opt-include=${opt-dir}/include
        --with-opt-lib
        --without-opt-lib=${opt-dir}/lib
        --with-make-prog
        --without-make-prog
        --srcdir=.
        --curdir
        --ruby=/usr/local/bin/$(RUBY_BASE_NAME)
        --help
        --download-dependencies
        --with-sqlcipher
        --without-sqlcipher
        --with-sqlcipher-dir
        --without-sqlcipher-dir
        --with-sqlcipher-include
        --without-sqlcipher-include
        --with-sqlcipher-lib
        --without-sqlcipher-lib
        --enable-system-libraries
        --disable-system-libraries
        --with-sqlcipher
        --without-sqlcipher
        --with-sqlcipher-dir
        --without-sqlcipher-dir
        --with-sqlcipher-include
        --without-sqlcipher-include
        --with-sqlcipher-lib
        --without-sqlcipher-lib
        --with-/usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc-dir
        --without-/usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc-dir
        --with-/usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc-include
--without-/usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc-include=${/usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc-dir}/include
        --with-/usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc-lib
--without-/usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc-lib=${/usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc-dir}/lib
        --with-/usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc-config
        --without-/usr/local/bundle/gems/sqlite3-1.5.3/ports/x86_64-linux-gnu/sqlite3/3.39.4/lib/pkgconfig/sqlite3.pc-config
        --with-pkg-config
        --without-pkg-config

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /usr/local/bundle/extensions/x86_64-linux/3.1.0/sqlite3-1.5.3/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /usr/local/bundle/gems/sqlite3-1.5.3 for inspection.
Results logged to /usr/local/bundle/extensions/x86_64-linux/3.1.0/sqlite3-1.5.3/gem_make.out

  /usr/local/lib/ruby/site_ruby/3.1.0/rubygems/ext/builder.rb:102:in `run'
  /usr/local/lib/ruby/site_ruby/3.1.0/rubygems/ext/ext_conf_builder.rb:28:in `build'
  /usr/local/lib/ruby/site_ruby/3.1.0/rubygems/ext/builder.rb:171:in `build_extension'
  /usr/local/lib/ruby/site_ruby/3.1.0/rubygems/ext/builder.rb:205:in `block in build_extensions'
  /usr/local/lib/ruby/site_ruby/3.1.0/rubygems/ext/builder.rb:202:in `each'
  /usr/local/lib/ruby/site_ruby/3.1.0/rubygems/ext/builder.rb:202:in `build_extensions'
  /usr/local/lib/ruby/site_ruby/3.1.0/rubygems/installer.rb:843:in `build_extensions'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/rubygems_gem_installer.rb:72:in `build_extensions'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/rubygems_gem_installer.rb:28:in `install'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/source/rubygems.rb:207:in `install'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/installer/gem_installer.rb:54:in `install'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/installer/parallel_installer.rb:186:in `do_install'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/installer/parallel_installer.rb:177:in `block in worker_pool'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/worker.rb:62:in `apply_func'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/worker.rb:57:in `block in process_queue'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/worker.rb:54:in `loop'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/worker.rb:54:in `process_queue'
  /usr/local/bundle/gems/bundler-2.3.23/lib/bundler/worker.rb:91:in `block (2 levels) in create_threads'

An error occurred while installing sqlite3 (1.5.3), and Bundler cannot continue.

In Gemfile:
  sqlite3
ERROR: 5

Possibly this gem needs to add pkg-config as a dependency.

oliverklee avatar Oct 12 '22 08:10 oliverklee

Thanks for reporting this! I'll take a look later today.

Out of curiosity: is there a reason you're not using the precompiled native version?

flavorjones avatar Oct 12 '22 13:10 flavorjones

No - actually, I would prefer the precompiled version. :-)

I was doing only a bundle install (with the new version in the Gemfile.lock). Is there anything else I need to do to get the precompiled version?

oliverklee avatar Oct 12 '22 13:10 oliverklee

This is the PR with the change: https://github.com/braingourmets/crud/pull/425

You can run dip provision to check this on your machine (with Dip, Docker and Docker-Compose available).

oliverklee avatar Oct 12 '22 13:10 oliverklee

Thanks for your patience. I poked around a bit and here's what I found.

The vanilla "ruby" platform gem versions v1.5.3 and v1.5.2 behave identically when the target system does not have the pkg-config utility installed (i.e. apt install pkg-config). (The pkg-config gem isn't actually used if it's present! Issue created here.)

So there was no change introduced that I can see that would be causing this problem.

However, the thing we should look into is: why is this system not using the precompiled native version for x86_64-linux?

What is the output of bundle env on this system (please run it in the project directory)? I'm curious if perhaps you're using a version of bundler with bugs (there has been some functionality changes in the last few patch releases).

flavorjones avatar Oct 12 '22 17:10 flavorjones

This is the output of bundle env in the container:

## Environment
Bundler       2.3.23
  Platforms   ruby, x86_64-linux
Ruby          3.1.2p20 (2022-04-12 revision 4491bb740a9506d76391ac44bb2fe6e483fec952) [x86_64-linux]
  Full Path   /usr/local/bin/ruby
  Config Dir  /usr/local/etc
RubyGems      3.3.21
  Gem Home    /usr/local/bundle
  Gem Path    /app/tmp/cache/gem/ruby/3.1.0:/usr/local/lib/ruby/gems/3.1.0:/usr/local/bundle
  User Home   /root
  User Path   /app/tmp/cache/gem/ruby/3.1.0
  Bin Dir     /usr/local/bundle/bin
Tools         
  Git         2.30.2
  RVM         not installed
  rbenv       not installed
  chruby      not installed

Bundler Build Metadata

Built At          2022-10-05
Git SHA           250d9d485d
Released Version  true

Bundler settings

app_config
  Set via BUNDLE_APP_CONFIG: ".bundle"
jobs
  Set via BUNDLE_JOBS: 4
retry
  Set via BUNDLE_RETRY: 3
silence_root_warning
  Set via BUNDLE_SILENCE_ROOT_WARNING: true

Gemfile

Gemfile

# frozen_string_literal: true

source 'https://rubygems.org'

# Rails
gem 'rails', '6.1.7'

# The original asset pipeline for Rails [https://github.com/rails/sprockets-rails]
gem 'sprockets-rails'

# Use sqlite3 as the database for Active Record
gem 'sqlite3'

# Use Puma as the app server
gem 'puma'

# Use Haml for HTML templates (http://haml.info)
gem 'haml'
gem 'haml-rails'

# Use SCSS for stylesheets
gem 'cssbundling-rails'
gem 'jsbundling-rails'

# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
gem 'turbolinks', '~> 5'

# Reduces boot times through caching; required in config/boot.rb
gem 'bootsnap', require: false

# Temporarily needed for some other gem.
gem 'net-smtp'

group :development do
  # Provide better error pages
  gem 'better_errors'
  gem 'binding_of_caller'
  gem 'listen'
  gem 'rack-mini-profiler'
end

group :test do
  # Style checkers
  gem 'haml_lint', require: false
  gem 'psych'
  gem 'rails_best_practices', require: false
  gem 'reek', require: false
  gem 'rubocop', require: false
  gem 'rubocop-performance', require: false
  gem 'rubocop-rails', require: false
  gem 'rubocop-rake', require: false

  # Security checker
  gem 'bundler-audit', require: false
end

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    actioncable (6.1.7)
      actionpack (= 6.1.7)
      activesupport (= 6.1.7)
      nio4r (~> 2.0)
      websocket-driver (>= 0.6.1)
    actionmailbox (6.1.7)
      actionpack (= 6.1.7)
      activejob (= 6.1.7)
      activerecord (= 6.1.7)
      activestorage (= 6.1.7)
      activesupport (= 6.1.7)
      mail (>= 2.7.1)
    actionmailer (6.1.7)
      actionpack (= 6.1.7)
      actionview (= 6.1.7)
      activejob (= 6.1.7)
      activesupport (= 6.1.7)
      mail (~> 2.5, >= 2.5.4)
      rails-dom-testing (~> 2.0)
    actionpack (6.1.7)
      actionview (= 6.1.7)
      activesupport (= 6.1.7)
      rack (~> 2.0, >= 2.0.9)
      rack-test (>= 0.6.3)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.0, >= 1.2.0)
    actiontext (6.1.7)
      actionpack (= 6.1.7)
      activerecord (= 6.1.7)
      activestorage (= 6.1.7)
      activesupport (= 6.1.7)
      nokogiri (>= 1.8.5)
    actionview (6.1.7)
      activesupport (= 6.1.7)
      builder (~> 3.1)
      erubi (~> 1.4)
      rails-dom-testing (~> 2.0)
      rails-html-sanitizer (~> 1.1, >= 1.2.0)
    activejob (6.1.7)
      activesupport (= 6.1.7)
      globalid (>= 0.3.6)
    activemodel (6.1.7)
      activesupport (= 6.1.7)
    activerecord (6.1.7)
      activemodel (= 6.1.7)
      activesupport (= 6.1.7)
    activestorage (6.1.7)
      actionpack (= 6.1.7)
      activejob (= 6.1.7)
      activerecord (= 6.1.7)
      activesupport (= 6.1.7)
      marcel (~> 1.0)
      mini_mime (>= 1.1.0)
    activesupport (6.1.7)
      concurrent-ruby (~> 1.0, >= 1.0.2)
      i18n (>= 1.6, < 2)
      minitest (>= 5.1)
      tzinfo (~> 2.0)
      zeitwerk (~> 2.3)
    ast (2.4.2)
    better_errors (2.9.1)
      coderay (>= 1.0.0)
      erubi (>= 1.0.0)
      rack (>= 0.9.0)
    binding_of_caller (1.0.0)
      debug_inspector (>= 0.0.1)
    bootsnap (1.13.0)
      msgpack (~> 1.2)
    builder (3.2.4)
    bundler-audit (0.9.1)
      bundler (>= 1.2.0, < 3)
      thor (~> 1.0)
    code_analyzer (0.5.5)
      sexp_processor
    coderay (1.1.3)
    concurrent-ruby (1.1.10)
    crass (1.0.6)
    cssbundling-rails (1.1.1)
      railties (>= 6.0.0)
    debug_inspector (1.1.0)
    erubi (1.11.0)
    erubis (2.7.0)
    ffi (1.15.5)
    globalid (1.0.0)
      activesupport (>= 5.0)
    haml (6.0.6)
      temple (>= 0.8.2)
      thor
      tilt
    haml-rails (2.1.0)
      actionpack (>= 5.1)
      activesupport (>= 5.1)
      haml (>= 4.0.6)
      railties (>= 5.1)
    haml_lint (0.42.0)
      haml (>= 4.0, < 6.1)
      parallel (~> 1.10)
      rainbow
      rubocop (>= 0.50.0)
      sysexits (~> 1.1)
    i18n (1.12.0)
      concurrent-ruby (~> 1.0)
    jsbundling-rails (1.0.3)
      railties (>= 6.0.0)
    json (2.6.2)
    kwalify (0.7.2)
    listen (3.7.1)
      rb-fsevent (~> 0.10, >= 0.10.3)
      rb-inotify (~> 0.9, >= 0.9.10)
    loofah (2.19.0)
      crass (~> 1.0.2)
      nokogiri (>= 1.5.9)
    mail (2.7.1)
      mini_mime (>= 0.1.1)
    marcel (1.0.2)
    method_source (1.0.0)
    mini_mime (1.1.2)
    mini_portile2 (2.8.0)
    minitest (5.16.3)
    msgpack (1.5.6)
    net-protocol (0.1.3)
      timeout
    net-smtp (0.3.2)
      net-protocol
    nio4r (2.5.8)
    nokogiri (1.13.8)
      mini_portile2 (~> 2.8.0)
      racc (~> 1.4)
    parallel (1.22.1)
    parser (3.1.2.1)
      ast (~> 2.4.1)
    psych (4.0.6)
      stringio
    puma (5.6.5)
      nio4r (~> 2.0)
    racc (1.6.0)
    rack (2.2.4)
    rack-mini-profiler (3.0.0)
      rack (>= 1.2.0)
    rack-test (2.0.2)
      rack (>= 1.3)
    rails (6.1.7)
      actioncable (= 6.1.7)
      actionmailbox (= 6.1.7)
      actionmailer (= 6.1.7)
      actionpack (= 6.1.7)
      actiontext (= 6.1.7)
      actionview (= 6.1.7)
      activejob (= 6.1.7)
      activemodel (= 6.1.7)
      activerecord (= 6.1.7)
      activestorage (= 6.1.7)
      activesupport (= 6.1.7)
      bundler (>= 1.15.0)
      railties (= 6.1.7)
      sprockets-rails (>= 2.0.0)
    rails-dom-testing (2.0.3)
      activesupport (>= 4.2.0)
      nokogiri (>= 1.6)
    rails-html-sanitizer (1.4.3)
      loofah (~> 2.3)
    rails_best_practices (1.23.2)
      activesupport
      code_analyzer (~> 0.5.5)
      erubis
      i18n
      json
      require_all (~> 3.0)
      ruby-progressbar
    railties (6.1.7)
      actionpack (= 6.1.7)
      activesupport (= 6.1.7)
      method_source
      rake (>= 12.2)
      thor (~> 1.0)
    rainbow (3.1.1)
    rake (13.0.6)
    rb-fsevent (0.11.1)
    rb-inotify (0.10.1)
      ffi (~> 1.0)
    reek (6.1.1)
      kwalify (~> 0.7.0)
      parser (~> 3.1.0)
      rainbow (>= 2.0, < 4.0)
    regexp_parser (2.6.0)
    require_all (3.0.0)
    rexml (3.2.5)
    rubocop (1.36.0)
      json (~> 2.3)
      parallel (~> 1.10)
      parser (>= 3.1.2.1)
      rainbow (>= 2.2.2, < 4.0)
      regexp_parser (>= 1.8, < 3.0)
      rexml (>= 3.2.5, < 4.0)
      rubocop-ast (>= 1.20.1, < 2.0)
      ruby-progressbar (~> 1.7)
      unicode-display_width (>= 1.4.0, < 3.0)
    rubocop-ast (1.21.0)
      parser (>= 3.1.1.0)
    rubocop-performance (1.15.0)
      rubocop (>= 1.7.0, < 2.0)
      rubocop-ast (>= 0.4.0)
    rubocop-rails (2.16.1)
      activesupport (>= 4.2.0)
      rack (>= 1.1)
      rubocop (>= 1.33.0, < 2.0)
    rubocop-rake (0.6.0)
      rubocop (~> 1.0)
    ruby-progressbar (1.11.0)
    sexp_processor (4.16.1)
    sprockets (4.1.1)
      concurrent-ruby (~> 1.0)
      rack (> 1, < 3)
    sprockets-rails (3.4.2)
      actionpack (>= 5.2)
      activesupport (>= 5.2)
      sprockets (>= 3.0.0)
    sqlite3 (1.5.2)
      mini_portile2 (~> 2.8.0)
    stringio (3.0.2)
    sysexits (1.2.0)
    temple (0.8.2)
    thor (1.2.1)
    tilt (2.0.11)
    timeout (0.3.0)
    turbolinks (5.2.1)
      turbolinks-source (~> 5.2)
    turbolinks-source (5.2.0)
    tzinfo (2.0.5)
      concurrent-ruby (~> 1.0)
    unicode-display_width (2.3.0)
    websocket-driver (0.7.5)
      websocket-extensions (>= 0.1.0)
    websocket-extensions (0.1.5)
    zeitwerk (2.6.0)

PLATFORMS
  ruby

DEPENDENCIES
  better_errors
  binding_of_caller
  bootsnap
  bundler-audit
  cssbundling-rails
  haml
  haml-rails
  haml_lint
  jsbundling-rails
  listen
  net-smtp
  psych
  puma
  rack-mini-profiler
  rails (= 6.1.7)
  rails_best_practices
  reek
  rubocop
  rubocop-performance
  rubocop-rails
  rubocop-rake
  sprockets-rails
  sqlite3
  turbolinks (~> 5)

BUNDLED WITH
   2.3.23

oliverklee avatar Oct 12 '22 18:10 oliverklee

I've just tested some things. It turned out it's not the 1.5.2-to-1.5.3 switch that has caused the problem, but the update from Bundler 2.3.22 to 2.3.23.

oliverklee avatar Oct 12 '22 18:10 oliverklee

(I have no idea whether this is a bug in the sqlite gem or in Bundler.)

oliverklee avatar Oct 12 '22 18:10 oliverklee

OK, this sounds like it might be a bundler behavior change ... but regardless, let's get you unblocked.

Solution 1: use precompiled native gems

I imagine your Gemfile.lock is checked into source code control? If so, the easiest change here is to run this command in development:

bundle lock --add-platform x86_64-linux

and commit the updated Gemfile.lock to source code control. If you have macs as development machines you may need to repeat this for x86_64-darwin and/or arm64-darwin.

The resulting Gemfile.lock will have a PLATFORMS stanza that will include those platforms (currently it only contains ruby in your bundle env output above).

Explicitly adding those platforms will ensure that you get the precompiled native gems and avoid the pkg-config problem.

Solution 2: install pkg-config

The other option is to just make sure your container has the debian pkg-config package installed. You can add this to your build script:

apt update
apt install -y pkg-config

Let me know if either of these suggestions doesn't work?

flavorjones avatar Oct 12 '22 19:10 flavorjones

We've now worked around this by downgrading Bundler (which allowed us to install/update sqlite again).

How can I help to get this solved for the future (on the sqlite or bundler side)?

oliverklee avatar Oct 13 '22 09:10 oliverklee

There isn't a problem with the sqlite3 gem. You need pkg-config (the distro package) installed to install the "ruby" platform gem.

There may be a problem with bundler, but it's more likely that it is also functioning as designed and you need to evolve your Gemfile.lock in response to the changes introduced since Bundler 2.2.x in order to properly use native platform gems. I've instructed you how to do that above.

flavorjones avatar Oct 13 '22 11:10 flavorjones

I'm not at all denying that something went wrong here ... I'm just not sure there is anything that needs to be fixed upstream in this gem or in Bundler. I'm sorry for the difficulties!

flavorjones avatar Oct 13 '22 11:10 flavorjones