Maybe support PyPy using cpyext?

[PallHaraldsson]

Maybe this is helping:

https://doc.pypy.org/en/latest/release-v7.3.1.html

Conda Forge now supports PyPy as a python interpreter.

I'm not pushing for this (it's not high-priority, would only make faster, maybe more secure), I just noticed they use UTF-8 (and CPython does Unicode, still, differently), like Julia so might be a no-copy to/from Python if it's PyPy.

Even if we do not do this, their UTF-8 implementation is intriguing with optional O(1) indexing, into variable length UTF-8 (something Julia should consider doing, possibly PyPy [code] could help?).

https://doc.pypy.org/en/latest/faq.html#what-about-numpy-numpypy-micronumpy

Depending on status of this, PyPy could be safer: https://doc.pypy.org/en/latest/sandbox.html

I think they also use ordered Dicts, not sure (either way should work, with it we need to consider changing PyCall, need to anyway, see my other issue on that): https://twitter.com/pypyproject/status/942070907715182592?lang=en

[stevengj]

We can't support PyPy, because it doesn't support the API of the libpython (CPython) library. This has nothing to do with UTF-8.

[yuyichao]

PyPy does support part of the CPython API through cpyext (enough to run numpy though I couldn't find the reference right now...) as well as constructing your own API using cffi in a similar fashion to @cfunction. Either of these could work in the short term. (though of course none of the details mentioned in the top post including string encoding, dict semantics, sandboxing has much to do with this...)

There's actually a proposal to unify the API better https://pythoncapi.readthedocs.io/roadmap.html https://hpy.readthedocs.io/en/latest/. I wasn't following the development closely but I though there's already an implementation of the API for both cpython and pypy. Depending on the stability of the API right now, porting to this should be the way forward.

[stevengj]

It still seems different enough that we’d need a complete rewrite.

I’m not sure I see the point. Most libraries that people want to call from Julia are still for cpython, no?

[yuyichao]

The cpyext one should not need much or any rewrite, and the hpy API rewrite would be needed in the long term.

As for the need, quite a few python libraries that I would find useful, including scipy, sympy and matplotlib do appear to work with pypy. So while I'm neutral regarding whether it worth starting to support pypy/hpy now, the number of supported python module with the performance benefit could justify someone to want to use PyCall with pypy.

[yuyichao]

The cpyext one should not need much or any rewrite

Which also means that someone should probably try to just install pycall and point the python executable to pypy to have a try... I don't want to break my local setup so I'm not going to test it any time soon......