ray
ray copied to clipboard
[Serve] Support sync function for multiplexing
Description
Currently @serve.multiplexed only support async def.
Eventually we can have
@serve.multiplexed
def load_model(mode_id:str):
return
@serve.deployment
class Model:
async def __call__(self, req):
model = await load_model(req.meta.model_id)
We can prioritize the feature till we gather enough customer signals.
Use case
No response
This P2 issue has seen no activity in the past 2 years. It will be closed in 2 weeks as part of ongoing cleanup efforts.
Please comment and remove the pending-cleanup label if you believe this issue should remain open.
Thanks for contributing to Ray!