serve
serve copied to clipboard
workflow manager async prediction response causes timeout potentially
Symptom: ab test sent out 11000 requests. The last inference request didn’t get response even though backend processed the request successfully. root cause: applyAsync causes the order of thread completion is unpredictable. Potentially it can cause an inference response will be timeout.