RCall Failing with rJava

[TheCedarPrince]

trafficstars

Hi JuliaInterop team,

Thanks for making a great package! That said, I am running into a rather odd issue that I am not sure how to surmount.

Error Description

I am trying to use a package within R called DatabaseConnector for SQL calls. Some other packages I am using also depend on this package. Whenever I try to use it, I get this error:

R> library('DatabaseConnector')
Error: package or namespace load failed for ‘DatabaseConnector’:
 .onLoad failed in loadNamespace() for 'rJava', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object '/home/cedarprince/.local/share/renv/cache/v5/R-4.0/x86_64-pc-linux-gnu/rJava/0.9-13/cfcca6464c6d145e28924800b5bd7f44/rJava/libs/rJava.so':
  libjvm.so: cannot open shared object file: No such file or directory

When I try to examine or load rJava, I get this problem as well:

R> library('rJava')
Error: package or namespace load failed for ‘rJava’:
 .onLoad failed in loadNamespace() for 'rJava', details:
  call: dyn.load(file, DLLpath = DLLpath, ...)
  error: unable to load shared object '/home/cedarprince/.local/share/renv/cache/v5/R-4.0/x86_64-pc-linux-gnu/rJava/0.9-13/cfcca6464c6d145e28924800b5bd7f44/rJava/libs/rJava.so':
  libjvm.so: cannot open shared object file: No such file or directory

Also, additionally, I have done this approach:

export LD_LIBRARY_PATH=/usr/lib/jvm/java-16-openjdk-16.0.0.0.36-1.rolling.fc32.x86_64/lib/server/
sudo R CMD javareconf
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:`R RHOME`/lib"

Which then hangs in the Julia REPL for a moment and then seg faults:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using RCall

R> library('DatabaseConnector')
[1]    2978888 segmentation fault (core dumped)  julia

It is odd as I have definitely installed these packages within R itself. Furthermore, the R code I have written has zero problems with rJava and everything works perfectly.

I have used both the R version provided by dnf (4.0.4) and I have compiled and built from source R-0.4.0.5 with the --enable-R-shlib. The error manifests across both installs. What is going on here?

Version Information

Here is my Julia Version information:

Julia Version 1.6.0
Commit f9720dc2eb (2021-03-24 12:55 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)
Environment:
  JULIA_EDITOR = nvim

and my R version information

R> R.version
               _
platform       x86_64-pc-linux-gnu
arch           x86_64
os             linux-gnu
system         x86_64, linux-gnu
status
major          4
minor          0.5
year           2021
month          03
day            31
svn rev        80133
language       R
version.string R version 4.0.5 (2021-03-31)
nickname       Shake and Throw

Here is my OS information:

          /:-------------:\          cedarprince@obsidian
       :-------------------::        --------------------
     :-----------/shhOHbmp---:\      OS: Fedora x86_64
   /-----------omMMMNNNMMD  ---:     Host: XPS 15 7590
  :-----------sMMMMNMNMP.    ---:    Kernel: 5.11.10-100.fc32.x86_64
 :-----------:MMMdP-------    ---\   Uptime: 1 day, 6 hours, 39 mins
,------------:MMMd--------    ---:   Shell: zsh
:------------:MMMd-------    .---:   Resolution: 1920x1080, 1080x1920, 1920x1080
:----    oNMMMMMMMMMNho     .----:   DE: Plasma 5.18.5
:--     .+shhhMMMmhhy++   .------/   WM Theme: Sweet-Dark-transparent
:-    -------:MMMd--------------:    CPU: Intel i7-9750H (12) @ 4.500GHz
:-   --------/MMMd-------------;     Memory: 6132MiB / 15637MiB
:-    ------/hMMMy------------:
:-- :dMNdhhdNMMNo------------;
:---:sdNMMMMNds:------------:
:------:://:-------------::
:---------------------://

[TheCedarPrince]

Ok! I am back! This is obviously not an optimal solution but I discovered that this seems to work with Julia 1.0! :tada: I was able to get everything working going back to that Julia version. However, I would like to get this working with Julia 1.6 - though I am on R 4.0.5, it still works with this old of a version. Thoughts as to what is going on?

[TheCedarPrince]

After further experimentation, I seem to have got this mostly working with Julia 1.6 and R 4.0.5! Here are the exact steps I took:

1. Install and build R from source. I used these directions found here from Rstudio support: https://support.rstudio.com/hc/en-us/articles/218004217-Building-R-from-Source
   • I tweaked the configuration step slightly where instead of providing a direct prefix, I did ./configure --enable-R-shlib
2. Install Julia 1.6 from downloads here: https://julialang.org/downloads/
3. Install Java. As I was using fedora, I installed java-16-openjdk-16.0.0.0.36-1.rolling.fc32.x86_64 via dnf.
4. Export your LD_LIBRARY_PATH to the location of the libjvm.so shared object file.
   • In my case, my export looked like this: export LD_LIBRARY_PATH=/usr/lib/jvm/java-16-openjdk-16.0.0.0.36-1.rolling.fc32.x86_64/lib/server/
5. Use R to reconfigure its Java path by running sudo R CMD javareconf
6. Export your library path again to include the path to your RHOME variable: export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:R RHOME/lib"
7. Boot up Julia and set the environment variable R_HOME to where your R installation is located (the directory, not the binary). For me, this looked as follows: ENV["R_HOME"] = "/usr/local/lib64/R".
8. Install RCall via ]add RCall
9. Build RCall via ]build RCall
10. Use RCall and you should be set!

This worked for me and got around the java issues - mostly. When I run an R object string (i.e. R"....."), my R code successfully works. Strangely, in the R Julia REPL mode, it fails with a Java error again here:

R> result <- renderTranslateQuerySql(connection, sql, cdm ="main")
Error in .jcall("java/lang/Class", "Ljava/lang/Class;", "forName", cl,  :
  RcallMethod: cannot determine object class
Error in .jfindClass(as.character(class)) :
  Java Exception <no description because toString() failed>.jfindClass(as.character(class))new("jobjRef", jobj = <poi
nter: 0x2364e88>, jclass = "java/lang/Throwable")

Yet, that same line of R works perfectly in the native R REPL.

However, for the most part I am quite content and happy. This gives me full interop with R as far as I can tell and I can use renv for managing my R packages. If anyone has any thoughts as to where this failure is propagating from, please let me know!

[Non-Contradiction]

Just see this issue, and have some idea about it (but not sure if it is in the correct direction).

From the code and the error message, it seems that that some package for SQL is involved, and the error message is that the some object class cannot be determined. I'm wondering whether this is caused by that the SQL package defines some S3 or S4 class in R, and also the difference between embedding R and interactive R (for example, some package related to S3 or S4 will be implicitly loaded by interactive R but not in embedding or non-interactive R, if I remember correctly, package methods may be one of them), or the difference in package loading (for example, there may also be some difference in R package loading/attaching). I guess one quick way is to check whether the code runs non-interactively. Or maybe we can check whether it works if we load packages like methods manually.

[Non-Contradiction]

By the way, one irrelevant question, is the R building from source and the julia official binary working with JuliaCall? Thanks!

[TheCedarPrince]

Hey @Non-Contradiction,

Thanks for the response. I am quite naive about R so bear with me as I ask some follow up questions:

• Could you give me some exact steps to try regarding running the code non-interactively? What do you mean by that? I have already been able to successfully execute the same R code in 4.0.5 and in the R"..." macro syntax. When I run R line-by-line in the Julia R REPL, it fails there with the above Java message.
• How should I go about loading in packages like methods? library('methods')?

Thanks for helping me! Hopefully we can track down where this is coming from!

[TheCedarPrince]

Oh also, the official Julia binary (Version 1.6.0) and R (4.0.5) built from source are working together! Here is a screenshot:

[Non-Contradiction]

• The method of running R non-interactively means running R in "batch mode", and can be found in https://stackoverflow.com/questions/18306362/run-r-script-from-command-line. So basically we have the R code in some file and use things like rscript TheRCodeFile.R in terminal.
• Yes, we can load the methods package by library(methods).

[TheCedarPrince]

Hey @Non-Contradiction - here are the results:

1. I ran the Rscript command on my script that I had written that is partially failing in RCall. It works successfully without error.
2. I ran the library('methods') command both in the native R REPL and Julia-R REPL - no errors there either.

What should I do next? Was that helpful at all?

[Non-Contradiction]

Does library(methods) solve the previous problem? Now I think I was on a wrong track though... Is there any example to reproduce the "cannot determine class" error?

[TheCedarPrince]

Hey @Non-Contradiction - I tried running library('methods') and it didn't change anything.

Also, here is some R code to try reproducing on your end:

install.packages(c("SqlRender", "DatabaseConnector", "remotes"))
remotes::install_github("ohdsi/Eunomia", ref = "v1.0.0")

connectionDetails <- Eunomia::getEunomiaConnectionDetails()
library(DatabaseConnector)
connection <- connect(connectionDetails)
sql <- "SELECT *
FROM @cdm.condition_occurrence
LIMIT 1;"
result <- renderTranslateQuerySql(connection, sql, cdm = "main")

Let me know if I can provide anything else. :)

[Non-Contradiction]

I try the code snippet and have some interesting observations:

• On my laptop windows, all good.
• On the Redhat7 of my university cluster,
  • The code runs fine in plain R.
  • In julia and the RCall R REPL, the code leads to the original libjvm.so issue, and I cannot build R from source on the university cluster. Instead, I try to use JuliaCall from R, and see what happens.
  • In a clean R session, if I do library(JuliaCall); julia_setup(); julia_console(); ## which opens the julia console in R. And in the julia console, I go into the R console (JuliaCall will use RCall by default), and code like connectionDetails <- Eunomia::getEunomiaConnectionDetails() will cause segfault. I guess the segfault here may correspond to the segfault in the original issue when you set the corresponding library path.
  • In a clean R session, still load the JuliaCall, but do not use julia_console() at first, instead, do julia_command('R"connectionDetails <- Eunomia::getEunomiaConnectionDetails()"'), the code runs fine for me mostly, but sometimes with a warning Rjava.init.warning: stack size reduced from unlimited to 9957581 bytes after JVM initialization. If the code runs without warning, then the Rjava init successfully, and if you goes into the julia_console() and then the RCall R REPL, then everything should be fine except the current issue we are discussing that .jfindClass(as.character(class)) : Java Exception <no description because toString() failed>. If the Rjava.init.warning happens, then the Rjava init with some problem, and in the julia_console(), every RCall related command will lead to stack usage problem. However, one interesting thing is that you can still call RCall command in julia_command without any problem like julia_command('R"1+1"').
  • In a clean R session, if we run everything using julia_command('R"....."'), that is to say, RCall command nested in julia_command, then all the code runs okay without any problems, except the sometimes Rjava.init.warning.

So in sum, I somehow can reproduce all (?) the issues we have here using subtle combinations of JuliaCall, RCall, Julia REPL, R REPL, etc. Ideally, all these combinations should be equivalent, but this is clearly not the case here.

An initial observation for these combinations is that all the methods inside a Julia REPL fail, including the segfault error (if we don't trigger Rjava initiation in advance) or .jfindClass(as.character(class)) : Java Exception <no description because toString() failed> (if Rjava initiation is done successfully). So I'm wondering whether Julia REPL is behind all these issues.

Also, all the methods not inside a Julia REPL run without any problem including plain R and julia_command('R"...."'), except the RJava.init.warning.

[TheCedarPrince]

I am glad you were able to reproduce these strange errors. I am not sure if there is anything to be done as the work around of putting everything into the R"..." macros syntax provides a workaround for these errors. Thanks for confirming things on my end @Non-Contradiction - feel free to close this issue or keep it open for more experimenting. :smile: