Support for PyCall?

Hi,

Nice work!

Anything that gets me away from Python is a good thing. However, increasingly it has become the lingua franca - or perhaps, "English of the Internet" - of scientific computing, in that most people barely ever consider using anything else and you have to use it, whether you like it or not. I have regrettably embedded a python interpreter in a C++ application recrntly for calculating filter co-efficients, simply because even in C/C++, the effort of actually finding an individual package (that is probably old and not well-maintained) for each type of filter and then integrating it with my code was outweighted by the convenience of having access to a multitude of algorithms that are only really made available as pip packages, even if that's a horrible solution.

To that end, I have just tried pycall and admittedly after only a few minutes roadtesting, so far surprisingly seems it "just works", which is especially remarkable given that Python rarely "just works" with itself. So Iwas wondering how much effort it would likely be to see if numpy/libraries making use of numpy could interface with Numo arrays through the Ruby side.

I know Numpy 2.0 changed its API, so that might introduce some friction, and also that it works pretty well through things like Pybind11 with storage backed by Eigen, xtensor, or even a plain vector, so I haven't yet dived into the specifics but as far as I know, Numpy should have reasonable support for a python-side array with dimensions and strides that wraps some buffer it doesn't manage itself. I ended up having a play and knocked up a basic bit of pure-ruby to send values from Numo into NumPy here, which is far from sufficient but let me have a play without rebuilding.

As a side-note, looks like the backend is implemented in C, but if you'd consider switching to C++ (in part at least), the new std::mdspan might make life easier, and is set to replace a lot of code for working with matrices, multi-dimensional arrays, etc. In implementation terms it's not much more than a non-owning pointer with a length and plenty of compile-time safety features attached to it, but when multiple different algorithms or units of code that need to talk to each other make use of it, it makes things a good deal safer and easier to work with, leaving the creater of that array responsibility for management of lifecycle, allocations, etc., and if you take an input and an output span then you just get an iterator to run over for performing computations.

Thanks

Hello. I will write some background information.

NArray is a fairly old library - version 0.3.0 of NArray was released in 1999. (You and I registered it on GitHub in 2013)

• https://speakerdeck.com/masa16tanaka/progress-of-ruby-numo-numerical-computing-for-ruby?slide=11
• https://github.com/masa16/narray/graphs/contributors

Many people have created a number of matrix calculation libraries for Ruby, but none of them have ever surpassed the performance of NArray.

PyCall, on the other hand, is a slightly newer library. PyCall's author, mrkn, had the idea of using an efficient memory arrangement called MemoryView as an intermediate to interchange matrices of various classes in Ruby, which was introduced in Ruby 3.0.

• https://docs.ruby-lang.org/en/master/memory_view_md.html
• [JA] https://tech.speee.jp/entry/2020/12/24/093131

However, implementation has been slow. I think this is the first time I've heard of the idea of using Python's mechanism for interfacing with other matrix libraries, which is on the Python side.

For converting between NArray and C++, there is a library called Numo.hpp.

• https://github.com/ankane/numo.hpp

This is the historical background of the project. For my part, I think we should add a committer or reviewer to the project to take over Numo::NArray for the next generation and promote a bit more team development.

Translated with DeepL.com (free version)