git sourced gems

[BrentWheeldon]

Hi there,

We are having an issue with gems in our gemfile that are sourced from git. We have set up a sample repo at https://github.com/vsanta/warble_test which exhibits this problem when you run warble && java -jar warble_test.war -S rake -T. The output is:

→ java -jar warble_test.war -S rake -T
Bundler::GitError: https://github.com/gregbell/active_admin.git (at master) is not yet checked out. Run `bundle install` first.
  load_spec_files at /var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/jruby1985128161926521553extract/gems/gems/bundler-1.5.2/lib/bundler/source/git.rb:179
      local_specs at /var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/jruby1985128161926521553extract/gems/gems/bundler-1.5.2/lib/bundler/source/path.rb:86
            specs at /var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/jruby1985128161926521553extract/gems/gems/bundler-1.5.2/lib/bundler/source/git.rb:150
  __materialize__ at /var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/jruby1985128161926521553extract/gems/gems/bundler-1.5.2/lib/bundler/lazy_specification.rb:53
      materialize at /var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/jruby1985128161926521553extract/gems/gems/bundler-1.5.2/lib/bundler/spec_set.rb:88
             map! at org/jruby/RubyArray.java:2445
      materialize at /var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/jruby1985128161926521553extract/gems/gems/bundler-1.5.2/lib/bundler/spec_set.rb:85
            specs at /var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/jruby1985128161926521553extract/gems/gems/bundler-1.5.2/lib/bundler/definition.rb:133
        specs_for at /var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/jruby1985128161926521553extract/gems/gems/bundler-1.5.2/lib/bundler/definition.rb:178
            setup at /var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/jruby1985128161926521553extract/gems/gems/bundler-1.5.2/lib/bundler/runtime.rb:13
            setup at /var/folders/44/zr3txl1n45b23kd2kgx1xqq00000gn/T/jruby1985128161926521553extract/gems/gems/bundler-1.5.2/lib/bundler.rb:126
           (root) at <script>:8
error: org.jruby.embed.EvalFailedException: (GitError) https://github.com/gregbell/active_admin.git (at master) is not yet checked out. Run `bundle install` first.

We're using:

• jruby 1.7.9
• bundler 1.5.2
• warbler HEAD
• rvm 1.25.15 and rbenv 0.4.0

We've spent some time looking in bundler and warbler code, but haven't worked out where this problem is coming from. We noticed https://github.com/jruby/warbler/pull/231/ seemed to relate to the same issue, but it doesn't work for us.

Any tips would be greatly appreciated!

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/1446592-git-sourced-gems?utm_campaign=plugin&utm_content=tracker%2F136961&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F136961&utm_medium=issues&utm_source=github).

[BrentWheeldon]

So adding config.override_gem_home = false to our warbler config fixed this. We're not sure why https://github.com/jruby/warbler/pull/231's test coverage is green, it seems from our experience it should be red.

[LFDM]

I tried to reproduce this a little earlier but failed. :( Did you run bundle immediately before you created the war? Currently don't really look completely through the way warbler is organizing gems...

[jwinter]

@BrentWheeldon The reason setting config.override_gem_home = false fixes this issue is that it makes the assignment on this line an or-equals (||=): https://github.com/jruby/warbler/blob/master/lib/warbler/templates/war.erb#L3 . WarMain.java exits with those ENV vars, so they're already set and it works.

I spent a good amount of time debugging this to try to figure out what this line should be: https://github.com/jruby/warbler/blob/master/lib/warbler/templates/war.erb#L3 but I can't figure it out. Using the value of of File.expand_path('../../WEB-INF', __FILE__) never works on that line, so if ENV['GEM_HOME'] is assigned to it, it fails.

I've got a failing spec in this branch if anyone has the time to look at it: https://github.com/jwinter/warbler/blob/failing-spec-for-git-gems-override-gem-home/spec/warbler/bundler_spec.rb#L86

[richievos]

+1 have repro'd this when attempting to upgrade from bjeanes fork of warbler (from #136). With that version -S rake -T works fine. With latest it doesn't see git sourced gems.

@BrentWheeldon thanks for your workaround, I can confirm that also fixed it for me

[dipolesource]

I've been struggling with the same problem and found that my Bundler version mattered. I was able to build a working war with Bundler 1.6.2 while the same commit would build a failing war with Bundler 1.6.1.

Anyone else care to give the Bundler version a try to confirm?

[amalagaura]

@dipolesource I am having this problem and I am using Bundler 1.6.2

I also have it working with config.override_gem_home = false

[rtyler]

I believe the problem is relatively easily solved with a Bundler.require in the config.ru.

E.g.

require 'rubygems'
require 'bundler'

Bundler.require

require 'my-git-based-gem'

map "/" do
  run Stuff
end

This is what solves using git-based gems inside of my bundled up war file.

[jesseproudman]

Any further solution on this?

[jesseproudman]

We're seeing this issue, except the provided fix doesn't resolve the issue:

config.override_gem_home = false

Here is our warble.rb config file:

Warbler::Config.new do |config|
  config.features = %w(runnable compiled)

  # Build what?
  config.dirs = %w(app config db lib log public tmp)
  config.includes = FileList["config.ru", "Rakefile"]

  # Array of regular expressions matching relative paths in gems to be
  # excluded from the war. Defaults to empty, but you can set it like
  # below, which excludes test files.
  config.gem_excludes = [/^(test|spec|docs|fixtures)\//]

  # When set to true, Warbler will override the value of ENV['GEM_HOME'] even it
  # has already been set. When set to false it will use any existing value of
  # GEM_HOME if it is set.
  # See: https://github.com/jruby/warbler/issues/235
  config.override_gem_home = false

  # Value of RAILS_ENV for the webapp -- default as shown below
  config.webxml.rails.env = ENV['RAILS_ENV'] || 'production'

  # Application booter to use, one of :rack, :rails, or :merb (autodetected by default)
  config.webxml.booter = :rails

  # Set JRuby to run in 1.9 mode.
  config.webxml.jruby.compat.version = "1.9"

  # Name the result
  config.jar_name = 'app'
end

Version:

• Warbler 1.4.4
• Bundler 1.7.12
• Jruby 1.7.18

Ripping my hair out on this one...

[jesseproudman]

And for what it's worth, our missing git based gem is actually missing from the war:

dhcp69:jproudman-auth jproudman$ jar tfv auth.war | grep "rails_sql_views"
dhcp69:jproudman-auth jproudman$ 

[jesseproudman]

Also, we've tested this both with RVM and rbenv to the same results.

[jesseproudman]

So, following the instructions here (https://github.com/jruby/warbler/issues/165#issuecomment-60734455) and using:

bundle package --all

We were able to get the the war to build and operate successfully.

Something isn't right here still w/ how warbler is interacting with bundler and paths for git gems.

Let me know what I can do to help.

[jesseproudman]

Gremlins in the system. I just left the office and tried again and it seems to be working.

[tillsc]

I can confirm the bundle package --all workaround does the trick. But it duplicates the WAR file size by storing the gems under 'vendor/cache' (by bundler) and 'gems' (by warbler). I found another workaround for that excluding '.gem' files in 'vendor/cache' in config/warble.rb:

config.excludes = FileList[..., "vendor/cache/*.gem", ...]

That seems to work because warbler stores rubygems dependencies as '.gem' files while it stores :git dependencies as unpacked gems.

Clearly this is not a good solution but it seems to work for me.