Retiring jruby support

[julia-pu]

TLDR: We are looking to retire support for jruby, and would like to hear some feedback.

Currently we are experiencing some issues while attempting to support jruby:

• updated versions of bundler(2.2.x) is breaking the jruby tests
• jruby test flakiness

We are interested in knowing if there are people currently using LicenseFinder in jruby. We haven't seen any activities/PRs around jruby in LicenseFinder for a while. Our plan is to listen for feedback for 30 days, after which we will decide whether or not to remove support for jruby. For now, we are going to remove the jruby test suite since it is currently having some problems with the updated version of bundler.

We are requesting feedback on whether or not LicenseFinder is being used in jruby. If so, are there people willing to pitch in and help fix problems with jruby?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[headius]

We would like to help support JRuby support here. What is involved? What is the issue with continuing support?

[xtreme-shane-lattanzio]

Hey @headius ! The reason for this is that there is a lot of overhead for us with jruby and we are not sure how much of our user base actually uses jruby. In this case, bundler updated to 2.2.x and it started to break ONLY jruby in our pipelines. We did not see the value in making the time investment for investigating it and have been talking internally for a long time about dropping jruby support and felt this may be a good time.

However, we do not want to leave any users in the dark due to this hence making this issue to get a feel for jruby usage in the community. If you would like to dig in to these issues, I would be happy to add jruby back to our pipelines. Until the issue is fixed, we will probably not gate LF releases on jruby tests though unless we do get the community support showing us the value. The docs here did say that there should be better jruby support but we just saw it broken.

I added the pipeline back here: https://norsk.cf-app.com/teams/main/pipelines/LicenseFinder/jobs/ruby-jruby-9.2.14.0/builds/1

Give it a look when you have a chance!

[headius]

I took a quick look and it appears all ten failures are the same issue, something preventing it from finding the Gemfile.

Hopefully I can try to run this locally without a lot of setup... if I can reproduce we should be able to fix this. It might be a Bundler bug, though... they do not test on JRuby as extensively as on CRuby and keep regressing as a result.

[headius]

Well I got as far as this, following along with the CI job, and I am not sure how to proceed. Can you give me a 30 second walkthrough for how I can reproduce the build errors locally?

[] ~/projects/LicenseFinder $ bundle exec rake features
LicenseFinder::GoDep: is not installed
LicenseFinder::Glide: is not installed
LicenseFinder::Gvt: is not installed
LicenseFinder::Govendor: is not installed
LicenseFinder::Dep: is not installed
LicenseFinder::Bower: is not installed
LicenseFinder::Gradle: is not installed
LicenseFinder::Rebar: is not installed
LicenseFinder::Nuget: is not installed
LicenseFinder::Mix: (elixir) is not installed
LicenseFinder::Sbt: is not installed
LicenseFinder::Cargo: is not installed
LicenseFinder::Dotnet: is not installed
LicenseFinder::Composer: is not installed

[xtreme-shane-lattanzio]

Hey! If you run the LF code in the LF docker image all of those package managers should be installed so I would try with that: https://hub.docker.com/r/licensefinder/license_finder/

[headius]

@xtreme-shane-lattanzio I will give that a try, thanks!

[headius]

I have managed to reproduce locally using the Docker image, thanks!

[headius]

I only got the following two failures locally for rake features:

  1) Bundler Dependencies works with git dependency running outside of the bundle
     Failure/Error:
       ::Bundler.with_unbundled_env do
         # Fake home to avoid polluting real home with git checkout
         ENV['HOME'] = LicenseFinder::TestingDSL::Paths.projects.to_s
         # Make running license_finder pick up the correct implementation
         ENV['RUBYOPT'] = "-I#{LicenseFinder::TestingDSL::Paths.root}/lib"
         # Allow the main executable to be found
         ENV['PATH'] = "#{LicenseFinder::TestingDSL::Paths.root}/bin:#{ENV['PATH']}"
         LicenseFinder::TestingDSL::GitBundlerProject.create
         puts 'bundler project created'
         bundler_developer.run_license_finder
     
     NoMethodError:
       undefined method `with_unbundled_env' for Bundler:Module
       Did you mean?  with_clean_env
     # ./features/features/package_managers/bundler_spec.rb:25:in `block in <main>'

  2) Conda are shown in reports
     Failure/Error: LicenseFinder::TestingDSL::CondaProject.create
     
     RuntimeError:
       Command failed: `bash -c 'source /root/miniconda3/etc/profile.d/conda.sh && conda env create -f environment.yml'`
       output: Collecting package metadata (repodata.json): ...working... /root/miniconda3/etc/profile.d/conda.sh: line 81: 11288 Killed                  "$CONDA_EXE" $_CE_M $_CE_CONDA "$cmd" "$@"
       exit: 137
     # ./features/support/testing_dsl.rb:794:in `run'
     # ./features/support/testing_dsl.rb:719:in `block in shell_out'
     # ./features/support/testing_dsl.rb:719:in `shell_out'
     # ./features/support/testing_dsl.rb:140:in `install'
     # ./features/support/testing_dsl.rb:84:in `create'
     # ./features/features/package_managers/conda_spec.rb:13:in `block in <main>'

The first issue may be due to the bundler shipped with JRuby being too old and not upgrading when on your CI. That method was added in bundler 2.2.0, and JRuby 9.2.14.0 ships 1.17.3. We are fixing that for upcoming releases.

The second failure seems to be a failed launch of "conda"? Any help there?

[headius]

JRuby master appears to pass all tests once I upgrade Bundler, including the "conda" tests. I am getting results with JRuby 9.2.14.0 now.

[headius]

A tip for speeding up the tests on JRuby: set JRUBY_OPTS env to --dev which will reduce optimization and improve startup time (these feature tests launch a lot of subprocesses).

[headius]

Locally I get 9 failures, all within the reporting, and most of them appear to be the same gemfile error. I suspect this is related to the older RubyGems shipped with 9.2.14.0 since JRuby master is green.

[xtreme-shane-lattanzio]

Hey @headius ! Thanks for continuing to look into this. I wouldn't worry about the conda tests for now. It could just be how your environment is set up and unrelated to jruby. For now I am more worried about the gemfile issue.

I think adding the JRUBY_OPTS flag is a great tip so we will look into that. Other than that, is there any way to fix the gems error or do we just need to wait for a new release of jruby?

[headius]

I had to refocus on some other JRuby tasks but this is still on my radar. It is not clear to me why these are failing yet but it maybe be a bug in the combination of bundler and our jar-dependencies library, which is used to handle JRuby gems with dependencies on published Java libraries. I need to isolate one test and see why the generation of the dummy app fails.