xoscar icon indicating copy to clipboard operation
xoscar copied to clipboard

Support fury as an optional serializer

Open codingl2k1 opened this issue 1 year ago • 3 comments

What do these changes do?

Related issue number

import xorbits
import time
from xorbits.remote import spawn
from xoscar.serialization.core import FurySerializer
from dataclasses import dataclass
from typing import List

@dataclass
class Foo:
    f1: int
    f2: int
    f3: float
    f4: int
    f5: int
    f6: List[int]


FurySerializer.register(Foo)

import logging

logging.basicConfig(level=logging.DEBUG)

if __name__ == "__main__":
    xorbits.init(config={"third_party_modules":["t1"]})


    def f1(x):
        return x


    foos = [Foo(f1=5000, f2=2 ** 32, f3=1 / 3, f4=100, f5=200, f6=[11, 1234]) for i in range(500)]

    start = time.time()
    results = []
    for _ in range(100):
        r1 = spawn(f1, foos)
        results.append(r1)

    xorbits.run(results)

    print(f"cost: {time.time() - start}")

This test code may be crash when fury deserializing, but if it does not crash, the performance is better than Mars default serializers.

Use fury(if not crash): 9.764425039291382 Mars default: 13.590802669525146

Fixes #xxxx

Check code requirements

  • [ ] tests added / passed (if needed)
  • [ ] Ensure all linting tests pass

codingl2k1 avatar Sep 13 '23 10:09 codingl2k1

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (2a041ae) 89.12% compared to head (d43d8f4) 88.98%.

Files Patch % Lines
python/xoscar/serialization/pyfury.py 83.33% 1 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #78      +/-   ##
==========================================
- Coverage   89.12%   88.98%   -0.15%     
==========================================
  Files          48       48              
  Lines        4010     4012       +2     
  Branches      764      764              
==========================================
- Hits         3574     3570       -4     
- Misses        351      356       +5     
- Partials       85       86       +1     
Flag Coverage Δ
unittests 88.80% <83.33%> (-0.15%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Sep 13 '23 10:09 codecov[bot]

Is there any things I can help for this PR?

chaokunyang avatar Nov 18 '23 13:11 chaokunyang

Is there any things I can help for this PR?

fury Python is hard to use, could you help fix the crash issue?

codingl2k1 avatar Nov 22 '23 05:11 codingl2k1