sglang icon indicating copy to clipboard operation
sglang copied to clipboard

Add retry to `http_request` if request timed out (ConnectionResetError)

Open Gintasz opened this issue 9 months ago • 0 comments

I was doing run_batch with 50-1000 batch items, connection to a remote server, num_threads 10-100 and I noticed ConnectionResetError is sometimes thrown by this call in http_request:

resp = urllib.request.urlopen(req, data=data, cafile=verify)

I think retrying while the server is still running (not crashed) is reasonable. Otherwise run_batch gets stuck and never finishes.

Gintasz avatar May 12 '24 07:05 Gintasz