bug: Bentoml mounted FastAPI app doesn't apply timeout value

[jtong99]

Describe the bug

Hi, I am running Bentoml and mount with FastAPI application.

app = FastAPI()

@bentoml.service(
    name="ai_service",
    resources={
        "cpu": CPU_THREADS,
        "memory": MEMORY,
    },
    traffic={"timeout": 3600}
)
@bentoml.asgi_app(app, path="/api")

@app.post('/v1/test')
async def ai_api(self, request: Request):

Even though I added timeout value to bentoml and run the service with timeout bentoml serve service:svc --timeout=3600, the API /v1/test only has 60s timeout. Is there anyway to make the API route from FastAPI which is mounted to Bentoml can accept the timeout value?

Thanks

To reproduce

No response

Expected behavior

The mounted FastAPI service accepted timeout value from Bentoml

Environment

Bentoml: latest

[frostming]

I can't reproduce using the latest version, here is the testing code:

import asyncio

import bentoml
import fastapi

app = fastapi.FastAPI()


@app.post("/greet")
async def greet():
    await asyncio.sleep(120)
    return "hello world"


@bentoml.service(traffic={"timeout": 360})
@bentoml.asgi_app(app)
class MyService:
    pass

Serve with bentoml serve and request with curl -XPOST http://localhost:3000/greet The response returns after around 120s

[jtong99]

Actually, it's about multiple APIs comes and it's timeout on processing after 60s, I thought it was about timeout variable but I checked the bentoml library and found it's because of this line at /bentoml/_internal/server/http/traffic.py:

loop.call_later(self.timeout, self._set_timer_out, waiter)

After I commented out this line, the issue is gone. The TimeoutMiddleware did not read the environment variable timeout, correct? Is there anyway to control this?

This is the error from bentoml regarding above error:

{"error":"Not able to process the request in 60.0 seconds"}

[frostming]

After I commented out this line, the issue is gone. The TimeoutMiddleware did not read the environment variable timeout, correct? Is there anyway to control this?

When you comment out this line, the middleware never times out.

self.timeout is retrieved from the config value, you can add some debug prints to verify the value.