Serverless generator incorrectly handles exceptions

[p0wx]

We run runpod serverless docker on our server for debugging. We have a handler function that looks like this:

async def handler(job):
    assert False, "My assert"
    yield "Hello"

We make a request like this that will invoke assert False, "My assert".

$ curl --request POST   --url http://localhost:8000/runsync   --header "accept: application/json"   --header "content-type: application/json"   --data '{...}'
Internal Server Error

We expected instead to get something like:

{"id":...,"status":"FAILED","error":"My assert"}

In server logs:

ERROR  | test-e8a0ec12-fee5-48c9-85fd-b77d6ed3753b | My assert
ERROR:    Exception in ASGI application
Traceback (most recent call last):
...
  File "/usr/local/lib/python3.10/dist-packages/runpod/serverless/modules/rp_fastapi.py", line 305, in _sim_runsync
    job_output['output'].append(stream_output["output"])
KeyError: 'output'

In serverless/modules/rp_fastapi.py

    async def _sim_runsync(self, job_request: DefaultRequest) -> JobOutput:
        """ Development endpoint to simulate runsync behavior. """
        assigned_job_id = f"test-{uuid.uuid4()}"
        job = TestJob(id=assigned_job_id, input=job_request.input)
        if is_generator(self.config["handler"]):
            generator_output = run_job_generator(self.config["handler"], job.__dict__)
            job_output = {"output": []}
            async for stream_output in generator_output:
                job_output['output'].append(stream_output["output"])
        else:
            job_output = await run_job(self.config["handler"], job.__dict__)
        if job_output.get('error', None):
            return jsonable_encoder({
                "id": job.id,
                "status": "FAILED",
                "error": job_output

Here if is_generator and stream_output is like {"error": ...} then line stream_output["output"] raises an error. So instead of error response from the server we get 500 internal server error.

Versions

• OS: Ubuntu 19.10
• Python: 3.11.8
• runpod (python): 1.6.2