"No such file or directory" when running `post_install_hook.sh`

[sschuberth]

Repeating a bit of background information from this Slack conversation:

I'm setting up a Kotlin / Gradle project (with a non-GraalVM JDK) in which I'd like to use TruffleRuby to run code from a Ruby Gem to be installed. My dependencies include

"org.graalvm.polyglot:polyglot:23.1.2"
"org.graalvm.polyglot:ruby:23.1.2"

and my Kotlin code looks like

Context.newBuilder().allowAllAccess(true).build().use { context ->
    context.eval("ruby", "Gem.install('licensee')")
}

However, this gives me

Caused by: org.graalvm.polyglot.PolyglotException: OpenSSL is not available. Install OpenSSL and rebuild Ruby (preferred) or use non-HTTPS sources

After an Internet search and reading through https://github.com/oracle/truffleruby/blob/master/doc/user/installing-libssl.md, I was looking for the post_install_hook.sh script on my system, which apparently gets installed at

~/.cache/org.graalvm.polyglot/ruby/ruby-home/710802544723712c59e4720b88bab3e008b5ef6c027bf4df856b9a50ac6cde761f38bc1d35dbd4a966ad7d58d8bfb3d6861b9652b03a4805428e4cb2d16d5d7f/lib/truffle/post_install_hook.sh

However, running that script (from the base directory with the long hash) yields

$ ./lib/truffle/post_install_hook.sh
Recompiling the OpenSSL C extension (against the installed libssl)
./lib/truffle/post_install_hook.sh: line 20: cd: src/main/c/openssl: No such file or directory

So apparently, the installation seems to be incomplete, or is looking for files in the wrong place.

Internal issue: GR-59808

[eregon]

Thanks for the report.

Writing down some thoughts on this: So at least one issue here is we'd need to ship src/main/c/openssl for the embedding-via-maven/gradle use case too. And we should make it easy/easier to run that when using the Context API, or at least document how. But the problem is lib/truffle/post_install_hook.sh needs a truffleruby executable and we don't ship it on Maven Central currently. So we would need to include that too. I'm not sure how that would work though because there won't be a librubyvm.so nor a jvm/ under the ruby home, so that truffleruby executable won't find java/libjvm. We'd need the caller Java process using the Context API to pass the java home or path to libjvm or so, e.g. via some env var. I don't think this information is passed currently.

We could also convert that script to a Ruby file, but that would not help, because recompiling openssl means running truffleruby extconf.rb in src/main/c/openssl and so we still need a truffleruby executable. Also we don't ship org.graalvm.ruby.launcher/org.truffleruby.launcher/RubyLauncher (the Java code specific to the launcher) to Maven Central currently (could be done though).

I think for now it should be possible to workaround by installing a TruffleRuby standalone and using that to recompile openssl, and copy the result over in the cache (~/.cache/org.graalvm.polyglot/ruby/ruby-home/...).

Another solution would be to vendor libssl, but that is problematic for multiple reasons, notably some C extensions do not work with that when they depend on a system package which depends on system libssl, loading multiple libssl in the same process feels brittle, and security-wise it's better to use the libssl from the operating system.

As a note, on Oracle Linux 7 there should be no need to recompile the openssl extension, because we ship it compiled against the Oracle Linux 7 system libssl.

One more possibility would be to reimplement Ruby OpenSSL on top of the Java security APIs. Unfortunately the Ruby OpenSSL API is so vast that reimplementing it on top of the Java security APIs seems a huge effort and unlikely to cover all the necessary parts (JRuby ships BouncyCastle + jruby-openssl for this). However GraalPy did this and it seems to work well (but different OpenSSL API in Python of course).

[eregon]

Something GraalPy does is shipping the Java module for the launcher code and using/generating a small Bash launcher script: https://github.com/oracle/graalpython/blob/release/graal-vm/23.1/graalpython/lib-graalpython/modules/standalone/main.py#L122 That seems the easiest solution.

And then we could document running the post-install hook using the maven exec plugin or similar.

[djberg96]

Similar error, different cause. I was trying to install truffleruby 24.1.1 on a Ubuntu 24 image, and hit this:

==> Installing truffleruby-24.1.1...
-> ./lib/truffle/post_install_hook.sh

BUILD FAILED (Ubuntu 24.04 on x86_64 using ruby-build 20241105-9-g7ccb143c)

Looking at the log I saw this:

compiling ossl_x509store.c
linking shared-object openssl.so

Recompiling the Psych C extension (against the installed libyaml)
checking for yaml.h... no
yaml.h not found
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--without-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/root/.rbenv/versions/truffleruby-24.1.1/bin/truffleruby
	--with-libyaml-source-dir
	--without-libyaml-source-dir
	--with-libyaml-dir
	--without-libyaml-dir
	--with-libyaml-include
	--without-libyaml-include=${libyaml-dir}/include
	--with-libyaml-lib
	--without-libyaml-lib=${libyaml-dir}/lib

Contents of mkmf.log:
find_header: checking for yaml.h... -------------------- no

LD_LIBRARY_PATH=. "gcc -o conftest -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include/ruby/backward -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include -I.    -O3 -fno-fast-math -ggdb3 -Werror=implicit-function-declaration -Wno-int-conversion -Wno-int-to-pointer-cast -Wno-incompatible-pointer-types -Wno-format-extra-args conftest.c  -L.          -L/root/.rbenv/versions/truffleruby-24.1.1/lib/cext -Wl,-rpath,/root/.rbenv/versions/truffleruby-24.1.1/lib/cext -ltrufflerubytrampoline"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: int main(int argc, char **argv)
4: {
5:   return !!argv[argc];
6: }
/* end */

LD_LIBRARY_PATH=. "gcc -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include/ruby/backward -I/root/.rbenv/versions/truffleruby-24.1.1/lib/cext/include -I.    -O3 -fno-fast-math -ggdb3 -Werror=implicit-function-declaration -Wno-int-conversion -Wno-int-to-pointer-cast -Wno-incompatible-pointer-types -Wno-format-extra-args   -c conftest.c"
conftest.c:3:10: fatal error: yaml.h: No such file or directory
    3 | #include <yaml.h>
      |          ^~~~~~~~
compilation terminated.
Process failed: #<Process::Status: pid 630 exit 1>
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <yaml.h>
/* end */

--------------------

external command failed with status 1

The solution was to explicitly add libyaml-dev as one of the installed packages.

[andrykonchin]

Nice work!

To be fair the dependencies (and libyaml as well) are listed in the README document and mentioned in the Installing TruffleRuby one.

[sschuberth]

To be fair the dependencies (and libyaml as well) are listed in the README document and mentioned in the Installing TruffleRuby one.

True, but I guess that's not something you can expect end-users of an application that leverages TruffleRuby to install Gems at runtime to follow...

[eregon]

We are thinking/planning to make this easier by vendoring libssl & libyaml. This should be quite helpful for embedding TruffleRuby (just pull the jars from Maven Central and done), and also as a side effect make it easier to install the TruffleRuby standalone by needing no post-install step or native dependencies.

I would like to keep this issue focused on embedding so I'll hide that comment by default as it is rather unrelated (it's about installing the truffleruby standalone, not embedding).

[sschuberth]

This should be quite helpful for embedding TruffleRuby (just pull the jars from Maven Central and done), and also as a side effect make it easier to install the TruffleRuby standalone by needing no post-install step or native dependencies.

That's awesome! It would make TruffleRuby way more accessible, which to me is one of the biggest problems currently with Graal Polyglot languages. The technology is great, but it's too cumbersome to set up e.g. in an existing Gradle project.