MLServer
MLServer copied to clipboard
Add cancelled requests back to the worker queue
Following #1234, a future enhancement is to - when a worker dies - add the cancelled requests back to the queue for another worker to pick up.
Originally posted by @sakoush in https://github.com/SeldonIO/MLServer/pull/1234#discussion_r1243766344