Problem with native extension compilation when using a git source

[schneems]

Hello! I've got a bit of a weird issue. When I install this gem on Heroku via rubygems it works fine.

source "https://rubygems.org"
ruby "3.1.3"

gem "seven_zip_ruby", github: "masamitsu-murase/seven_zip_ruby"

However when I install it via a git dependency it compiles the shared object to a location that causes problems.

Reproduction

Put this in the Gemfile:

source "https://rubygems.org"
ruby "3.1.3"

gem "seven_zip_ruby", github: "masamitsu-murase/seven_zip_ruby"

When I install this on Heroku the shared object file ends up in:

$ heroku run bash
~ $  find / -iname "7z.so"
# ...
/app/vendor/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/x86_64-linux/seven_zip_ruby/7z.so

This is a problem because this entire /app/vendor/ruby-3.1.3 directory is replaced on every deploy.

First deploy

So what happens is that on the first deploy Ruby is downloaded to /app/vendor/ruby-3.1.3 and then gems are installed where seven_zip_ruby creates /app/vendor/ruby-3.1.3/lib/ruby/site_ruby/3.1.0/x86_64-linux/seven_zip_ruby/7z.so and the deploy is fine. No problems.

Second deploy

The next deploy, Ruby is downloaded to /app/vendor/ruby-3.1.3 which overw-writes the shared object file. However Bundler sees that the gem is already installed so it won't try to recompile it. The release is successful but then at runtime the library cannot be loaded:

$ heroku run "bundle exec ruby -e 'require %Q{seven_zip_ruby}; puts %Q{worked}'"
/app/vendor/bundle/ruby/3.1.0/gems/seven_zip_ruby-1.3.0/lib/seven_zip_ruby.rb:12:in `find_external_lib_dir': Failed to find 7z.dll or 7z.so (RuntimeError)
        from /tmp/build_2d5b5659/vendor/bundle/ruby/3.1.0/gems/seven_zip_ruby-1.3.0/lib/seven_zip_ruby.rb:17:in `<module:SevenZipRuby>'

Thoughts

I don't know why this only happens when using a git source via bundler and only with this gem. This line is likely the reason that the gem is installed in that location https://github.com/masamitsu-murase/seven_zip_ruby/blob/fc296796bcf00a487a3d25dec3b0964158af6046/ext/seven_zip_ruby/extconf.rb#L7. I do not know what location bundler expects you to place shared objects while compiling. I work at Heroku for the past ~10 years and this is the only gem that I've seen that attempts to install to this location. I don't know where other gems typically install to.

I also don't know why we are getting different locations when using a rubygems source versus git source.

FWIW when I install via gem "seven_zip_ruby" (without git) the shared objects end up in these directories:

/app/vendor/bundle/ruby/3.1.0/extensions/x86_64-linux/3.1.0/seven_zip_ruby-1.3.0/seven_zip_ruby/7z.so
/app/vendor/bundle/ruby/3.1.0/gems/seven_zip_ruby-1.3.0/lib/seven_zip_ruby/7z.so

I think the directory that the gem installs to needs to be updated in extconf.rb but unfortunately I don't know what that directory should be. If anyone in the community knows. I would appreciate a comment.

[tramuntanal]

It seems to be standard to use the extensions dir inside the given gems dir for the current Ruby version. See:

> $LOAD_PATH
["/home/oliver/prog/decidim/apps/clean-app/lib",            
...
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/date-3.3.3/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/extensions/x86_64-linux/3.1.0/date-3.3.3",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/mini_mime-1.1.2/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/marcel-1.0.2/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/activerecord-6.1.7/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/activemodel-6.1.7/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/activejob-6.1.7/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/globalid-1.0.0/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/actioncable-6.1.7/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/websocket-driver-0.7.5/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/extensions/x86_64-linux/3.1.0/websocket-driver-0.7.5",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/websocket-extensions-0.1.5/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/nio4r-2.5.8/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/extensions/x86_64-linux/3.1.0/nio4r-2.5.8",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/actionpack-6.1.7/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/rack-test-2.0.2/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/rack-2.2.5/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/rails-html-sanitizer-1.4.3/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/loofah-2.3.1/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/crass-1.0.6/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/rails-dom-testing-2.0.3/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/nokogiri-1.13.10-x86_64-linux/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/racc-1.6.2/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/extensions/x86_64-linux/3.1.0/racc-1.6.2",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/mini_portile2-2.8.1/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/erubi-1.12.0/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/builder-3.2.4/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/activesupport-6.1.7/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/zeitwerk-2.6.6/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/tzinfo-2.0.5/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/minitest-5.16.3/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/i18n-1.12.0/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/rake-13.0.6/lib",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/site_ruby/3.1.0",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/site_ruby/3.1.0/x86_64-linux",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/site_ruby",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/vendor_ruby/3.1.0",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/vendor_ruby/3.1.0/x86_64-linux",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/vendor_ruby",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/3.1.0",
 "/home/oliver/.rbenv/versions/3.1.3/lib/ruby/3.1.0/x86_64-linux",

many gems use the .rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/extensions directory which suggests to be the convention.

[eregon]

Normally this is all handled by RubyGems, a gem's extconf.rb should not manually copy the .so anywhere, just build the .so and let RubyGems do the rest.

[andrewhamon]

@schneems I think this will be fixed by my PR: https://github.com/masamitsu-murase/seven_zip_ruby/pull/43

Its still weird to be manually copying the .so as @eregon says, but at least my PR puts it in the gem directory instead of the site ruby directory.