Publish `org.graalvm.ruby.launcher` as a Maven artifact

[sschuberth]

The background is this Slack conversation with @eregon: I'd like to use TruffleRuby to programmatically run CocoaPods (which is written in Ruby) from a Kotlin/JVM application. It seems the easiest way to do that would be by depending on a Maven artifact that includes org.truffleruby.launcher.RubyLauncher from my app, and then use that entry point to execute CocoaPods.

However, there currently is no Maven artifact that includes the org.truffleruby.launcher.RubyLauncher class. Thus my ask is to publish org.graalvm.ruby.launcher to Maven Central as well.

[sschuberth]

Looks like the GraalPy team does something similar already: https://mvnrepository.com/artifact/org.graalvm.python/python-launcher. I guess it makes sense to align here in terms of user experience.

[eregon]

Do you have an example CocoaPods command (and necessary files if any) that you would like to run so we can test if having that jar on Maven is enough or e.g. if CocoaPods needs a bin/ruby launcher to exist?

[sschuberth]

Some example commands are pod --version or pod repo add-cdn trunk https://cdn.cocoapods.org --allow-root.

[eregon]

FWIW I tried to install cocoapods with TruffleRuby jars on classpath and:

$ $ mx ruby_from_classpath --skip-build -S gem i cocoapods
Fetching nanaimo-0.4.0.gem
Fetching colored2-3.1.2.gem
Fetching claide-1.1.0.gem
Fetching CFPropertyList-3.0.7.gem
Fetching atomos-0.1.3.gem
Fetching xcodeproj-1.27.0.gem
Fetching ruby-macho-2.5.1.gem
Fetching nap-1.1.0.gem
Fetching molinillo-0.8.0.gem
Fetching gh_inspector-1.1.3.gem
Fetching fourflusher-2.3.1.gem
Fetching escape-0.0.4.gem
Fetching cocoapods-try-1.2.0.gem
Fetching netrc-0.11.0.gem
Fetching cocoapods-trunk-1.6.0.gem
Fetching cocoapods-search-1.0.1.gem
Fetching cocoapods-plugins-1.0.0.gem
Fetching cocoapods-downloader-2.1.gem
Fetching cocoapods-deintegrate-1.0.5.gem
Fetching ffi-1.17.2.gem
Fetching ethon-0.16.0.gem
Fetching typhoeus-1.4.1.gem
Fetching public_suffix-4.0.7.gem
Fetching fuzzy_match-2.0.4.gem
Fetching concurrent-ruby-1.3.5.gem
Fetching httpclient-2.9.0.gem
Fetching algoliasearch-1.27.5.gem
Fetching addressable-2.8.7.gem
Fetching tzinfo-2.0.6.gem
Fetching i18n-1.14.7.gem
Fetching connection_pool-2.5.3.gem
Fetching activesupport-7.2.2.1.gem
Fetching cocoapods-core-1.16.2.gem
Fetching cocoapods-1.16.2.gem
Successfully installed nanaimo-0.4.0
Successfully installed colored2-3.1.2
Successfully installed claide-1.1.0
Successfully installed CFPropertyList-3.0.7
Successfully installed atomos-0.1.3
Successfully installed xcodeproj-1.27.0
Successfully installed ruby-macho-2.5.1
Successfully installed nap-1.1.0
Successfully installed molinillo-0.8.0
Successfully installed gh_inspector-1.1.3
Successfully installed fourflusher-2.3.1
Successfully installed escape-0.0.4
Successfully installed cocoapods-try-1.2.0
Successfully installed netrc-0.11.0
Successfully installed cocoapods-trunk-1.6.0
Successfully installed cocoapods-search-1.0.1
Successfully installed cocoapods-plugins-1.0.0
Successfully installed cocoapods-downloader-2.1
Successfully installed cocoapods-deintegrate-1.0.5
Building native extensions. This could take a while...
ERROR:  Error installing cocoapods:
	ERROR: Failed to build gem native extension.

    current directory: /home/eregon/.cache/org.graalvm.polyglot/ruby/ruby-home/f1fe21562f21602e73bbd2b24ebc68e5028b9bc1a49fee5ca45f95d3724c528b827eb0db003d706fd5ae4c20e6032b616c1509c568f3c605c4a012a3657d48b2/lib/gems/gems/ffi-1.17.2/ext/ffi_c
/home/eregon/.cache/org.graalvm.polyglot/ruby/ruby-home/f1fe21562f21602e73bbd2b24ebc68e5028b9bc1a49fee5ca45f95d3724c528b827eb0db003d706fd5ae4c20e6032b616c1509c568f3c605c4a012a3657d48b2/bin/truffleruby extconf.rb
extconf failedcommand /home/eregon/.cache/org.graalvm.polyglot/ruby/ruby-home/f1fe21562f21602e73bbd2b24ebc68e5028b9bc1a49fee5ca45f95d3724c528b827eb0db003d706fd5ae4c20e6032b616c1509c568f3c605c4a012a3657d48b2/bin/truffleruby does not exist in PATH but posix_spawnp found it!

Gem files will remain installed in /home/eregon/.cache/org.graalvm.polyglot/ruby/ruby-home/f1fe21562f21602e73bbd2b24ebc68e5028b9bc1a49fee5ca45f95d3724c528b827eb0db003d706fd5ae4c20e6032b616c1509c568f3c605c4a012a3657d48b2/lib/gems/gems/ffi-1.17.2 for inspection.
Results logged to /home/eregon/.cache/org.graalvm.polyglot/ruby/ruby-home/f1fe21562f21602e73bbd2b24ebc68e5028b9bc1a49fee5ca45f95d3724c528b827eb0db003d706fd5ae4c20e6032b616c1509c568f3c605c4a012a3657d48b2/lib/gems/extensions/x86_64-linux/3.3.7.5/ffi-1.17.2/gem_make.out

FAILED (pid 135481 exit 1): mx ruby_from_classpath --skip-build -S gem i cocoapods

The error means installing gems with native extensions needs a bin/ruby launcher.

The gems could be installed via a TruffleRuby standalone as an alternative.

[eregon]

Installing via a TruffleRuby standalone works, e.g. via cd $HOME/tmp/cocoapods-gem-home && GEM_HOME=$PWD gem i cocoapods. And then using it works:

$ GEM_HOME=$HOME/tmp/cocoapods-gem-home mx ruby_from_classpath --skip-build -S pod --version
1.16.2

pod (to show the help) and pod repo add-cdn trunk https://cdn.cocoapods.org --allow-root also work.

So at least these 3 pod commands don't need a bin/ruby launcher. but installing the gem needs it.

[sschuberth]

So at least these 3 pod commands don't need a bin/ruby launcher. but installing the gem needs it.

What "magic" does the bin/ruby launcher script do on top of running org.truffleruby.launcher.RubyLauncher to make the installation of Gems work? Some LD_LIBRARY stuff I guess?

[eregon]

What "magic" does the bin/ruby launcher script do on top of running org.truffleruby.launcher.RubyLauncher to make the installation of Gems work? Some LD_LIBRARY stuff I guess?

No LD_LIBRARY and no magic, simply the existence of a bin/ruby executable, and its path is known from inside Ruby code with RbConfig.ruby or other RbConfig values like RbConfig::CONFIG["bindir"]. And gem installation of gems with extensions relies on the existence of that (to run the gem's extconf.rb).

If pulling only Maven jars it's only jars, there is no native ruby launcher/executable. We could pack it inside some jar and extract the native ruby launcher somewhere, but it wouldn't find a JDK to run on. One idea could be to maybe set some env var from the process using the Context API so that subprocesses would find the same JDK, but that would need changes in that native launcher, currently it always expects a jvm at ../jvm. Next problem is then how to find the various TruffleRuby jars (the classpath/modulepath) needed to run, since those are known from Maven but not for subprocesses.

And of course it seems a bad idea to just ship the whole TruffleRuby JVM standalone including a JVM as a jar, as it would be an extra JVM and duplicate truffleruby jars on disk 😅 (could also be the TruffleRuby Native standalone in a jar but still huge and some sort of duplication).

Also installing gems is normally done with the bundle install or gem install commands, which means these executables should exist, and they rely on bin/ruby to exist too, and they are not necessarily called from a process using the Context API. So the solution for now is to use a TruffleRuby standalone (either Native or JVM standalone, both work fine for this) to install gems (not unlike say compiling a native library with gcc and using the compiled library at runtime only needs dlopen() but there might be no gcc at runtime).

[eregon]

Giving a little bit more thoughts on this, I think using RubyLauncher won't work for such a use case, because it always does System.exit(exitCode), which is natural for a launcher but in your case would mean exiting the entire Java process using it.

So for your case the way to go seems to use e.g. context.eval("ruby", "CODE"). And CODE would be whatever you want to do with pod, here is how the pod executable is defined: https://github.com/CocoaPods/CocoaPods/blob/master/bin/pod. That might call Ruby exit BTW, but at least that can be caught when using the Context API as a PolyglotException so that's not a problem (exit just raises a SystemExit exception in Ruby).

Also, all Java APIs in TruffleRuby are private, none are public. The only public API is the polyglot API (defined in org.graalvm.polyglot such as Context, etc). So RubyLauncher should only be used internally. For that reason I think we should actually not publish RubyLauncher to Maven, as it's the internals used by the native launcher and I don't see another valid use case at the moment. IOW, if you want a ruby/truffleruby launcher currently you need a TruffleRuby standalone.

It might become necessary to expose some of RubyLauncher on Maven when the challenges described in https://github.com/oracle/truffleruby/issues/3894#issuecomment-3028045014 are solved, but there is a lot more to figure out there, and we should then also carefully review it to make the smallest public API possible.

As you noticed, GraalPy publishes their launcher, but that's a different setting as they have their own Maven & Gradle plugins and GraalPy uses that launcher in these plugins for installing packages. There are no Maven & Gradle plugins for TruffleRuby currently.

[fniephaus]

I agree and am reverting https://github.com/oracle/truffleruby/commit/4735b8143f516c535ab2a4dbd39beab15e7d9d3e in https://github.com/oracle/truffleruby/pull/3917.

[sschuberth]

I think using RubyLauncher won't work for such a use case, because it always does System.exit(exitCode)

Good point. I guess that could easily worked around by naively moving all code from main to a public function, and then main just calls this function plus System.exit(exitCode). That should meet your later mentioned requirement "to make the smallest public API possible".

As you noticed, GraalPy publishes their launcher, but that's a different setting as they have their own Maven & Gradle plugins and GraalPy uses that launcher in these plugins for installing packages. There are no Maven & Gradle plugins for TruffleRuby currently.

Yeah, I'd very much wish for TruffleRuby to have those plugins, too, and conveniently manage Ruby dependencies from a Java / Kotlin Gradle project as mentioned for GraalPython here.

[sschuberth]

IOW, if you want a ruby/truffleruby launcher currently you need a TruffleRuby standalone.

So, for the time being, would it make sense to publish a Maven artifact that allows to install all of TruffleRuby standalone just via a JAR artifact that I can specify in my build system as a dependency?

[eregon]

That should meet your later mentioned requirement "to make the smallest public API possible".

For now we actually want to expose nothing of RubyLauncher, it's completely internal API and functionality.

The polyglot API is designed for embedding, launcher code is not designed to be called from embeddings (aka applications written in Java and calling to Ruby).

So, for the time being, would it make sense to publish a Maven artifact that allows to install all of TruffleRuby standalone just via a JAR artifact that I can specify in my build system as a dependency?

Do you mean to have the .tar.gz standalone inside a .jar (one can't just put the contents in the .jar directly either as that loses permissions, symlinks, etc)? I don't think that would work well and the .tar.gz are platform-specific. With a standalone you would anyway need to run a subprocess to install gems, etc. So might as well download the archive and extract it in your code/logic, rather than having a big .jar on Maven Central.

Maybe you mean to have some Java logic, published as a Maven artifact, to download & extract the standalone and return the extracted path or so? There is no current plan to do that, but you could make your own.

[sschuberth]

There is no current plan to do that, but you could make your own.

I see. I've anyway meanwhile filed https://github.com/oracle/truffleruby/issues/3918 which should pave the way for embedding TruffleRuby in a way so that you can also install / depend on Gems with native code.