serve icon indicating copy to clipboard operation
serve copied to clipboard

workflow manager async prediction response causes timeout potentially

Open lxning opened this issue 3 years ago • 0 comments

Symptom: ab test sent out 11000 requests. The last inference request didn’t get response even though backend processed the request successfully. root cause: applyAsync causes the order of thread completion is unpredictable. Potentially it can cause an inference response will be timeout.

lxning avatar May 11 '22 22:05 lxning