[Python] pyfury, loky, pickle performance

[jacktang]

Hello there,

Did you have some benchmarks on pyfury, loky and pickle? Thanks!

[chaokunyang]

We did some simple benchmarks with pickle before, loky is not benchmarked.

There are still many space for performance optimization in python serialization, we haven't do the benchmark thoroughly.

[jacktang]

@chaokunyang Thanks for the quick reply. I read the python code quickly, and does pyfury supports python function serialization now?

[chaokunyang]

It doesn't support, you can pass a callback to let cloudpickle handle it

[chaokunyang]

from dataclasses import dataclass
from typing import List

@dataclass
class Foo:
    f1: int
    f2: int
    f3: float
    f4: int
    f5: int
    f6: List[int]

foos = [Foo(f1=1000, f2=2**30, f3=1/3, f4=100, f5=200, f6=[11, 1234]) for i in range(1000)]

fury = pyfury.Fury(language=pyfury.Language.PYTHON)
fury.register_class(Foo)

FYI, for above cases, fury is 1.35x faster than pickle.

There are still many virtual method calls in fury python, which we don't optimize still .

[esafak]

@jacktang Is your question answered?

[chaokunyang]

@chaokunyang Thanks for the quick reply. I read the python code quickly, and does pyfury supports python function serialization now?

@jacktang It supports now. You can use pyfory as a drop-in replacement for pickle/cloudpickle. And fory has 3x higher compression ratio.

See document in https://fory.apache.org/docs/docs/guide/python_serialization#drop-in-replacement-for-picklecloudpickle