Implementing PyCall.rb-like interop between TruffleRuby and Graal.python

[eregon]

MRI (CRuby) has a library to call out to Python called pycall.rb.

It would be interesting to make this work in GraalVM with TruffleRuby + GraalPython, where we could likely have a much simpler system (no serialization, no need to integrate with the GC, etc), and potentially better performance.

If we can, we'd basically make all the good scientific computing/ML libraries from Python available to Ruby (well, those running on GraalPython), with better performance and better interop than is possible when stitching two different VMs.

Longer term, it might be worth exploring having a GraalVM-backend in the upstream repository https://github.com/mrkn/pycall.rb.

cc @fniephaus

[eregon]

I met the author of pycall.rb, @mrkn, at RubyKaigi 2019. I showed him a simple example with a loop with the addition either in Ruby or Python, and comparing that to pycall.rb on CRuby, and of course that example is very fast on GraalVM.

I could install and load both numpy and pandas on GraalPython.

With just a few lines of code, we could define PyCall.import_module as a Polyglot.eval and a good part of the rest would just work thanks to using similar methods to access Python objects from Ruby.

module PyCall
  def self.import_module(name)
    Polyglot.eval('python', "import #{name}\n#{name}")
  end
end

I tried to get https://github.com/mrkn/pycall.rb/blob/master/examples/sum_benchmarking.rb running, but was blocked on installing seaborn on GraalPython. Maybe that works now.

mrkn sounded positive to support GraalVM in pycall.rb, so I think we should aim for that and do a PR.

For this to be convenient for Rubyists, we'd probably want to have a way to install GraalVM CE, Ruby and Python with Ruby installers. I can take care of that. It could also be interesting to experiment with a polyglot native image containing Ruby & Python here to improve startup/warmup.

[eregon]

An interesting design decision here would be to try to get the pycall.rb API working without needing to wrap Python objects in Ruby objects. That would reduce overhead and simplify passing objects between the two languages. If this doesn't work yet for some way to access the Python object, it would be interesting to discuss it here or on the GraalVM Slack.

[eregon]

@timfel said the example in https://www.practicalai.io/using-scikit-learn-machine-learning-library-in-ruby-using-pycall/ should be easy to get working.