sglang
sglang copied to clipboard
Add retry to `http_request` if request timed out (ConnectionResetError)
I was doing run_batch
with 50-1000 batch items, connection to a remote server, num_threads
10-100 and I noticed ConnectionResetError
is sometimes thrown by this call in http_request
:
resp = urllib.request.urlopen(req, data=data, cafile=verify)
I think retrying while the server is still running (not crashed) is reasonable. Otherwise run_batch
gets stuck and never finishes.