improving jitpy

[yannche]

Hi,

I really love pypy and jitpy, but for my work, I need to pass complex arguments to pypy functions, not just ints or strings. So my solution is to convert everything into strings with pickle. Of course it a very dirty hack, but my code still runs many,many times faster than python :-)

It would be great if jitpy could handle all this directly, with some kind of decorator. Unfortunately I don't have enough python/pypy expertise to customize jitpy. Here is the code I use to send/receive pickled strings.

from jitpy import setup
setup('/opt/pypy-2.5.0-linux/bin')
from jitpy.wrapper import jittify
import pickle
from cffi import FFI

@jittify([str],str)
def func(_pypy_arg):
    import pickle
    from cffi import FFI
    ffi = FFI()
    args = pickle.loads( ffi.string(_pypy_arg) )

    ret = "Processing "+str(args)+" and we're done..."

    return ffi.new("char[]",pickle.dumps(ret))


ffi = FFI()
_ret = func( pickle.dumps(["go",4,"it !!!"]))
print pickle.loads( ffi.string(_ret) )

Best regards, Yann

[fijal]

Hi Yann.

As a matter of principle, jitpy is as simple as it gets, we want less not more magic happening. So I suggest you distribute your hacks as a separate package and not part of jitpy.

That said, maybe passing cffi structures would be a good idea

[lambdaq]

support simple tuple, list (single type array) or dict would be awesome.

[arigo]

There is a good reason for why lists and dicts can't be passed, but as fijal's latest comment said, it would be possible to extend this simple hack to pass cffi pointers, arrays, and structures. Changes done inside the arrays or structures would be visible to both sides, as CPython and PyPy run in the same process. We'd welcome a pull request that attempts this.

[arigo]

A ten-minutes demo: https://bitbucket.org/cffi/cffiembed/overview

[lambdaq]

@arigo I am kind of n00b to this,

There is a good reason for why lists and dicts can't be passed

but is it because lists are mutable both in size and each element?

What if we pass read only list or dict?

[arigo]

Yes, it's possible to give a list or dict as read-only (i.e. make a copy for the other side). But at that point it is similar to using pickle or marshal to serialize and deserialize the list or dict.