quart

quart copied to clipboard

Problem Context Our interservice Rest API calls started failing suddenly a few days back, so when we looked into this we found the error as ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')) initially so we thought this is related to our services not being able to full fill all the requests so we looked into the Kubernetes stats where we found that some of our pods are getting unavailable suddenly when we looked into pods details we found that

Reason: ERROR Exit code: 137

initially, we thought this is related to memory allocation to the pods but then we have allocated ample memory to these pods and the nodes hosting them. So we started digging deep then we found this on StackOverflow that process is sending this error code which is being displayed by the pod.

So we tested and found that when the application is not able to create more subprocess or threads it throws this exit code so we tested by running this in a simple API in quart and found that the quart application is throwing the 137 exit code

async def func():
    while True:
        subprocess.Popen(['ls'])
    return "OK", 200

@pgjones can you please help on this as we are stuck on this why quart might exit with this exit code. Is there a way to handle this in a quart?

I would guess this is Hypercorn trying to create workers - does it work if you specify only one worker?

@pgjones To clarify we are using Uvicorn instead of Hypercorn with our quart microservices. Below is the command being used to start the server.

uvicorn app.server:app --host 0.0.0.0 --port 8000 --no-access-log

we are not able to replicate the same issue on our staging and dev environments. and we cannot directly test this scenario on production.

Is there a way we can check how many workers uvicorn is creating inside kubernetes pods before restarting?

any update?

I think it is worth asking on Uvicorn's issues as I think the exit code is being raised by it.