ray
ray copied to clipboard
[core] handle unserializable user exception
If I run the following script, core worker would fail to serialized the RayError and crash
(credit to @alexeykudinkin )
from time import sleep
import ray
ray.init()
@ray.remote(num_cpus=1)
class Callee:
def bar(self):
from tenacity import retry, stop_after_attempt
@retry(stop=stop_after_attempt(1))
def failing_method():
raise ValueError("failed")
failing_method()
@ray.remote(num_cpus=1)
class Caller:
def __init__(self, h):
self.callee = h
def foo(self):
ref = self.callee.bar.remote()
ray.wait([ref])
callee = Callee.remote()
caller = Caller.remote(callee)
ray.wait([caller.foo.remote()])
sleep(1800)
This PR will fix this.
@jjyao Is there any existing tests on testing exceptions?
@jjyao Test and doc added. PTAL!
@jjyao Ready for review. PTAL