jruby installation broken since 0.17.3

[rolandoam]

Description

Reproduction steps

1. Install the latest jruby (e.g. rbenv install jruby-9.4.8.0)
2. Try to install the latest ruby-lsp: gem install ruby-lsp
3. It will fail building the native extension

Code snippet or error message

Building native extensions. This could take a while...
ERROR:  Error installing ruby-lsp:
	ERROR: Failed to build gem native extension.

    current directory: /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/gems/shared/gems/rbs-3.5.2/ext/rbs_extension
/Users/rolando/.rbenv/versions/jruby-9.4.8.0/bin/jruby -I /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib extconf.rb
checking for whether -std=gnu99 is accepted as CFLAGS... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/Users/rolando/.rbenv/versions/jruby-9.4.8.0/bin/jruby
RuntimeError: The compiler failed to generate an executable file.
You have to install development tools first.

         try_do at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:456
    try_compile at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:571
    with_werror at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:522
    try_compile at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:571
     try_cflags at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:635
  append_cflags at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:641
   checking_for at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:942
       postpone at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:350
           open at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:320
       postpone at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:350
           open at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:320
       postpone at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:346
   checking_for at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:941
  append_cflags at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:640
           each at org/jruby/RubyArray.java:1981
  append_cflags at /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/stdlib/mkmf.rb:639
         <main> at extconf.rb:3

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/gems/shared/extensions/universal-java-22/3.1.0/rbs-3.5.2/mkmf.log

mkmf.log:

" -o conftest -I/include/universal-java22 -I/Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/include/ruby/backward -I/Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib/ruby/include -I.  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE     -DTARGET_RT_MAC_CFM=0  -fno-omit-frame-pointer -fno-strict-aliasing  -fexceptions  conftest.c  -L. -L/Users/rolando/.rbenv/versions/jruby-9.4.8.0/lib    -arch arm64      "
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <ruby.h>
4: int main(int argc, char **argv)
5: {
6:   return 0;
7: }
/* end */

Last known version to build seems to be 0.17.2:

gem install ruby-lsp -v 0.17.2
Successfully installed ruby-lsp-0.17.2
Parsing documentation for ruby-lsp-0.17.2
Done installing documentation for ruby-lsp after 2 seconds
1 gem installed

[github-actions[bot]]

This issue is being marked as stale because there was no activity in the last 2 months

[mrckzgl]

Same error here. I figured out that -o conftest -I/include/universal-java22 is the command which is executed. But as one can see it does not include an executable, only command line options. Having gcc installed I figured that mkmf gem is (at least in this specific case) relying on the CC env variable to be present. So having export CC=gcc somewhere in a shell initialization script will get you further, but only to:

"gcc -o conftest -I/include/universal-java17 -I~/.rvm/rubies/jruby-9.4.8.0/lib/ruby/include/ruby/backward -I~/.rvm/rubies/jruby-9.4.8.0/lib/ruby/include -I.     -fno-omit-frame-pointer -fno-strict-aliasing  -fexceptions  conftest.c  -L. -L~/.rvm/rubies/jruby-9.4.8.0/lib    -m64      "
In file included from conftest.c:1:
~/.rvm/rubies/jruby-9.4.8.0/lib/ruby/include/ruby.h:1:2: error: #error JRuby does not support native extensions
    1 | #error JRuby does not support native extensions
      |  ^~~~~
In file included from conftest.c:3:
~/.rvm/rubies/jruby-9.4.8.0/lib/ruby/include/ruby.h:1:2: error: #error JRuby does not support native extensions
    1 | #error JRuby does not support native extensions
      |  ^~~~~
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <ruby.h>
4: int main(int argc, char **argv)
5: {
6:   return 0;
7: }
/* end */

So did ruby-lsp switch from not using a native extension to using one?

[vinistock]

We added a dependency on rbs otherwise there's no way to index core declarations from the language (e.g.: String, Integer and so on).

Unfortunately, rbs doesn't support JRuby yet (which we didn't realize when we introduced it). And Rubygems doesn't support conditional/optional runtime dependencies, so we have no way of saying that we can only depend on rbs for the mri platform.

To fix this, we are going to need to ensure that rbs can work with JRuby.

[headius]

A short term option would be a -java platform gem that removes the rbs dependency, but we also want to get rbs support for JRuby.

[github-actions[bot]]

This issue is being marked as stale because there was no activity in the last 2 months

[headius]

What can I do to help move this along?

[vinistock]

The team is working on making the RBS parser pure C (without depending on Ruby internals), which should make it significantly easier to support JRuby.

Once all of those changes are shipped, we can simply bump our rbs requirement to use a version compatible with JRuby.

I'll post updates here once the work is further along. We can only provide a complete experience if we can index core declarations, which requires rbs, so I really think shipping these improvements is the ideal long term solution.

[headius]

@vinistock That sounds like a great project! Please let me know if there's any way I can help. We would very much like to see full RBS support in JRuby, and I have some fun ideas for integrating it with other JVM languages callable from Ruby.

[strawgate]

I was really hoping I could just install an old version of the extension to use with a ruby project or set the branch setting to run the old version but it seems no matter what extension version I run it always tries to install the latest gem on init?

For example If I set branch to v0.17.2 when it initializes I still get:

Failed to setup the bundle: Command failed: gem install ruby-lsp ERROR: Error installing ruby-lsp: ERROR: Failed to build gem native extension. current directory: /usr/local/bundle/gems/rbs-3.9.2/ext/rbs_extension /opt/jruby/bin/jruby extconf.rb checking for whether -std=gnu99 is accepted as CFLAGS... *** extconf.rb failed *** Could not create Makefile due to some reason, probably lack of necessary libraries and/or headers. Check the mkmf.log file for more details. You may need configuration options. Provided configuration options: --with-opt-dir --without-opt-dir --with-opt-include --without-opt-include=${opt-dir}/include --with-opt-lib --without-opt-lib=${opt-dir}/lib --with-make-prog --without-make-prog --srcdir=. --curdir --ruby=/opt/jruby/bin/jruby RuntimeError: The compiler failed to generate an executable file. You have to install development tools first. try_do at /opt/jruby/lib/ruby/stdlib/mkmf.rb:456 try_compile at /opt/jruby/lib/ruby/stdlib...

[vinistock]

@strawgate that's because the Ruby LSP's executable has two modes of operation. It is first used to compose the special bundle under .ruby-lsp without bundle exec and then running the server begins with bundle exec ruby-lsp. We favour using the latest version of the gem for composing the bundle because there's coordination between the extension and server.

You can pin version that will be used to run the server, but indeed it's currently not possible to pin the version that will be installed globally for launching.

That said, I would really prefer expediting the improvements to rbs so that JRuby users can enjoy our full experience. v0.17.2 is very old and missing a number of features and improvements we made.

[strawgate]

@strawgate that's because the Ruby LSP's executable has two modes of operation. It is first used to compose the special bundle under .ruby-lsp without bundle exec and then running the server begins with bundle exec ruby-lsp. We favour using the latest version of the gem for composing the bundle because there's coordination between the extension and server.

You can pin version that will be used to run the server, but indeed it's currently not possible to pin the version that will be installed globally for launching.

That said, I would really prefer expediting the improvements to rbs so that JRuby users can enjoy our full experience. v0.17.2 is very old and missing a number of features and improvements we made.

Thanks for the info, I am really excited to use this extension for working with ruby so that I can do all my dev in a devcontainer. If I'm feeling adventurous perhaps I'll fork for JRuby support and, if I do, will follow-up here.

[headius]

We are starting to hear of users leaving JRuby because of the lack of ruby-lsp support. They aren't going back to CRuby... they're leaving Ruby altogether, because without JRuby's scalability, they can no longer use Ruby.

What can I do to help get this issue moving? I am highly motivated. 😳

[vinistock]

The pure C RBS parser has been merged to main. We're already using that parser in Sorbet, so no more CRuby VM dependencies exist.

I think the last step to supporting JRuby in RBS should be adding Java bindings? Something similar to what Prism does, so that the parser can be hooked up to JRuby.

After that, as long as the RBS gem that supports JRuby is installed, everything should work. The LSP itself has no native code or anything that wouldn't work on JRuby.

[headius]

Aha, I think I expected that the "pure C parser" RBS gem would be using something like FFI to call into the library, but I see now there is a minimal C extension.

That is what you mean by Java bindings, correct?

I'm thinking the best path forward would be to create an FFI binding for the library that will work anywhere the C extension does not, and later on if we need it for performance we could do a more direct binding in JRuby (which would actually still be using FFI under the covers anyway).

Once that is available, we can modify the gem to skip building the extension on JRuby, and it will fall back on FFI.

I'll try to get the process started this week! Thanks @vinistock!

[vinistock]

Yeah, FFI should do the trick. I'm not very familiar with native extensions for JRuby, so I don't know if you need some Java layer to glue things together, which is what I was thinking of when I mentioned bindings.

I appreciate your help pushing this forward! As soon as RBS supports JRuby, we can close this issue and finally add JRuby to our CI.

Also, when there's a working branch, you can verify if the LSP is working correctly by adding rbs to the Gemfile and pointing to the working branch.

[headius]

I've kicked off the process here: https://github.com/ruby/rbs/pull/2572