Support JRuby?

[postmodern]

trafficstars

I would like to use the async gems under JRuby, but the io-event gem contains C extensions.

$ gem install async
Fetching timers-4.3.5.gem
Fetching async-2.0.0.gem
Fetching io-event-1.0.9.gem
Fetching fiber-local-1.0.0.gem
Fetching console-1.16.2.gem
Successfully installed timers-4.3.5
Building native extensions. This could take a while...
ERROR:  Error installing async:
	ERROR: Failed to build gem native extension.

    current directory: /home/postmodern/.gem/jruby/3.1.0/gems/io-event-1.0.9/ext
/opt/rubies/jruby-9.4.0.0/bin/jruby -I /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib extconf.rb
checking for rb_ext_ractor_safe()... *** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/opt/rubies/jruby-9.4.0.0/bin/jruby
RuntimeError: The compiler failed to generate an executable file.
You have to install development tools first.

        try_do at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:456
     try_link0 at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:541
      try_link at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:556
      try_func at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:765
     have_func at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:1051
  checking_for at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:942
      postpone at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:350
          open at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:320
      postpone at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:350
          open at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:320
      postpone at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:346
  checking_for at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:941
     have_func at /opt/rubies/jruby-9.4.0.0/lib/ruby/stdlib/mkmf.rb:1050
        <main> at extconf.rb:36
... 15 levels...

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /home/postmodern/.gem/jruby/3.1.0/extensions/universal-java-17/3.1.0/io-event-1.0.9/mkmf.log

extconf failed, exit code 1

Gem files will remain installed in /home/postmodern/.gem/jruby/3.1.0/gems/io-event-1.0.9 for inspection.
Results logged to /home/postmodern/.gem/jruby/3.1.0/extensions/universal-java-17/3.1.0/io-event-1.0.9/gem_make.out

[ioquatix]

JRuby is missing too many things right now to work correctly with io-event and the fiber scheduler but it's on their roadmap.

• https://github.com/jruby/jruby/issues/7512
• https://github.com/jruby/jruby/issues/4748

[mperham]

I'd like to see a pure version of io-event which does not have a native extension at all. Can the current state of non-blocking IO in Ruby support this ideal?

Right now I have to install a compiler on the production machine to install the Async gem (console gem has the same problem in requiring the json gem).

[postmodern]

@mperham that could be accomplished by converting io-event and nio4r gems into pre-compiled gems for the common platforms, similar to how nokogiri and sqlite3 gems are now built.

[ioquatix]

I'd like to see a pure version of io-event which does not have a native extension at all. Can the current state of non-blocking IO in Ruby support this ideal?

If there is a way to bail out of native extension compilation when it is not possible, that would work, and it will fall back to the pure Ruby implementation using IO.select. extconf.rb, mkmf and rubygems leaves a lot to be desired in this regard.

See https://github.com/rubygems/rubygems/pull/6569 for the kind of generic support we need to "optionally compile this extension" without working around all the very specific, hard coded build processes that are implemented in rubygems. If you have time to support or augment this request some how, I'd greatly appreciate it.

Right now I have to install a compiler on the production machine to install the Async gem (console gem has the same problem in requiring the json gem).

Is there any reason why we can't avoid that too by falling back to a pure Ruby implementation?

pre-compiled gems for the common platforms

I understand the general concept, but io-event has lots of platform and version specific checks. It may be quite tricky to compile for specific kernel versions, etc.

[ioquatix]

Just as one other thought, is there any way to pre-compile and vendor these gems in an application itself?

[postmodern]

If there is a way to bail out of native extension compilation when it is not possible, that would work, and it will fall back to the pure Ruby implementation using IO.select. extconf.rb, mkmf and rubygems leaves a lot to be desired in this regard.

I discovered from reading RubyGems extension compiling code, that it will not check if a .so extension file has been created, if you set gemspec.extensions to a Rakefile or anything besides extconf.rb. This would allow you to gracefully exit from compiling extensions if an exception is raised by mkmf. This is the strategy that digest-crc uses to fallback to using it's pure-Ruby implementations if the C extension files cannot be required. It's hacky, but it works.

[ioquatix]

It's hacky, but it works.

This is what I'm trying to avoid [hacky] :p but definitely appreciate your suggestion :)

Oh, one more thing is that it requires me to add a dependency on rake... which I'd like to avoid if possible.

[mperham]

I had my own fork on nio4r where I ripped out the native extension. I'm trying to avoid the same here, maybe instead I should do the research on how to make native extensions optional. Sounds like a pretty killer blog post if anyone else wants to get to it before me.

[ioquatix]

It's probably possible to just do this:

RUBY_NATIVE_EXTENSIONS=no bundle install

and in extconf.rb:

return if ENV['RUBY_NATIVE_EXTENSIONS'] == "no"

[postmodern]

@ioquatix RubyGems extconf.rb builder will check if a Makefile was created and will attempt to execute it. If there's no Makefile, it will error out. However, if you can generate a dummy Makefile that does nothing, it will successfully install.

require 'mkmf'

create_makefile 'test'

$ gem install ./test-0.0.1.gem 
Building native extensions. This could take a while...
Successfully installed test-0.0.1
Parsing documentation for test-0.0.1
Installing ri documentation for test-0.0.1
Done installing documentation for test after 0 seconds
1 gem installed

[ioquatix]

@mperham would that work for you? It still seems like you'd need some developer tools installed.

[ioquatix]

cc @simi

[mperham]

It's an improvement, I'd still have to install make but that's far less than a full compiler suite.