llama-stack
llama-stack copied to clipboard
meta-llama
AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
System Info
OpenShift Version: 4.17.24 OpenShift AI Version: 2.22.1 (rhods-operator.2.22.1) LlamaStack Distribution image built from Version: 0.2.14 GPU Type: NVIDIA L40S
Information
- [x] The official example scripts
- [ ] My own modified scripts
🐛 Describe the bug
This issue occurs when guidellm is used as a client to test performance of the llama-stack distro. The problem occurs on all concurrencies.
Setup Process
Set up the vLLM backend server with llama-32-3b-instruct model. Deploy a LlamaStackDistribution custom resource connected to vLLM server via the VLLM_URL environment variable. Deploy a GuideLLM benchmark job that targets the LlamaStack service. Configure it to run 1 concurrent request for 60 seconds, using 256 input tokens and 128 output tokens per request. The target URL should be the LlamaStack OpenAI-compatible endpoint.
Guidellm job description:
apiVersion: batch/v1
kind: Job
metadata:
name: guidellm-vllm-metrics-test
namespace: bench
spec:
ttlSecondsAfterFinished: 86400
backoffLimit: 0
template:
metadata:
labels:
job-name: guidellm-vllm-metrics-test
spec:
restartPolicy: Never
initContainers:
- name: timestamp-generator
image: busybox
command: ["sh", "-c"]
args:
- |
TS=$(date +"%Y%m%d-%H%M%S");
echo $TS > /shared/timestamp;
echo "Generated timestamp: $TS"
volumeMounts:
- name: shared-data
mountPath: /shared
containers:
- name: benchmark
image: ghcr.io/ccamacho/bench:main
imagePullPolicy: Always
env:
- name: GUIDELLM__MAX_WORKER_PROCESSES
value: "1"
- name: GUIDELLM__STREAM
value: "false"
command: ["/bin/bash", "-c"]
args:
- |
echo "Running benchmark...";
export HF_TOKEN=$(cat /secrets/token);
export TRANSFORMERS_OFFLINE=0;
export HF_HUB_DISABLE_TELEMETRY=1;
export HF_HOME=/cache;
export TRANSFORMERS_CACHE=/cache;
export HOME=/cache;
TS=$(cat /shared/timestamp);
OUTPUT_FILE="/output/results-guidellm-${TS}.json";
guidellm benchmark \
--output-path $OUTPUT_FILE \
--target http://llamastack-distribution-v2-23-service.llamastack.svc.cluster.local:8321/v1/openai \
--model llama-32-3b-instruct \
--processor meta-llama/Llama-3.2-3B-Instruct \
--rate-type concurrent \
--rate "1.0" \
--max-seconds 60 \
--data "prompt_tokens=256,output_tokens=128";
echo "Benchmark complete. Results stored in $OUTPUT_FILE";
echo "done" > /shared/benchmark_complete;
echo "Signaled nvidia-smi monitor to stop"
volumeMounts:
- name: results-volume
mountPath: /output
- name: hf-secret
mountPath: /secrets
readOnly: true
- name: hf-cache
mountPath: /cache
- name: shared-data
mountPath: /shared
- name: sidecar
image: busybox
command: ["sh", "-c", "sleep infinity"]
volumeMounts:
- name: results-volume
mountPath: /output
volumes:
- name: results-volume
emptyDir: {}
- name: hf-secret
secret:
secretName: hf-token-secret
- name: hf-cache
emptyDir: {}
- name: shared-data
emptyDir: {}
Error logs
INFO: 10.130.4.90:52664 - "POST /v1/openai/v1/completions HTTP/1.1" 200 OK
[2m12:48:01.260[0m [35m[END][0m [2m/v1/openai/v1/completions[0m[0m [StatusCode.OK][0m (1409.73ms)
[2m12:48:01.275[0m [35m[START][0m [2m/v1/openai/v1/completions[0m
INFO 2025-08-13 12:48:01,329 __main__:183 server: Generator cancelled
+ Exception Group Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 77, in collapse_excgroups
| yield
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 271, in __call__
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
| await self.app(scope, receive, _send)
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 327, in __call__
| return await self.app(scope, receive, send_with_trace_id)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 369, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
| await route.handle(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
| await self.app(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
| await response(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
| with collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
| self.gen.throw(value)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
| await func()
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
| async for chunk in self.body_iterator:
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/utils/context.py", line 32, in wrapper
| item = await gen.__anext__()
| ^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 185, in sse_generator
| await event_gen.aclose()
| ^^^^^^^^^^^^^^^^
| AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 327, in __call__
return await self.app(scope, receive, send_with_trace_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 369, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
await route.handle(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
await self.app(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
await response(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
with collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
await func()
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
async for chunk in self.body_iterator:
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/utils/context.py", line 32, in wrapper
item = await gen.__anext__()
^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 185, in sse_generator
await event_gen.aclose()
^^^^^^^^^^^^^^^^
AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
ERROR: Exception in ASGI application
+ Exception Group Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 77, in collapse_excgroups
| yield
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 271, in __call__
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
| await super().__call__(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/applications.py", line 113, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
| await self.app(scope, receive, _send)
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 327, in __call__
| return await self.app(scope, receive, send_with_trace_id)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 369, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
| await route.handle(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
| await self.app(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
| await response(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
| with collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
| self.gen.throw(value)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
| await func()
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
| async for chunk in self.body_iterator:
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/utils/context.py", line 32, in wrapper
| item = await gen.__anext__()
| ^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 185, in sse_generator
| await event_gen.aclose()
| ^^^^^^^^^^^^^^^^
| AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/app-root/lib64/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/applications.py", line 113, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 327, in __call__
return await self.app(scope, receive, send_with_trace_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 369, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
await route.handle(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
await self.app(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
await response(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
with collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
await func()
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
async for chunk in self.body_iterator:
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/utils/context.py", line 32, in wrapper
item = await gen.__anext__()
^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/distribution/server/server.py", line 185, in sse_generator
await event_gen.aclose()
^^^^^^^^^^^^^^^^
AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
[2m12:48:01.334[0m [35m[END][0m [2m/v1/openai/v1/completions[0m[0m [StatusCode.OK][0m (58.95ms)
12:48:01.332 [37m[INFO] Generator cancelled[0m
Expected behavior
No error
We have seen this once before. i think it would be great if you could print the event_gen variable when this happens. It clearly is getting the wrong value here somehow.
LlamaStack Distribution image built from Version: 0.2.10
does this reproduce w/ 0.2.17?
Currently I am using a build with 0.2.10 version from here: quay.io/opendatahub/llama-stack:0178233dd63f022886ffd237874d9c9ccb8e892d
How can I get access to the newest build version?
Currently I am using a build with 0.2.10 version from here: quay.io/opendatahub/llama-stack:0178233dd63f022886ffd237874d9c9ccb8e892d
How can I get access to the newest build version?
one from https://hub.docker.com/u/llamastack
cc @bbrowning @franciscojavierarceo @leseb
Yes, the error is reproduced in the 0.2.17 llamastack version. I tested the latest tag from the downstream repository that include these versions:
llama_api_client 0.2.0
llama_stack 0.2.17
llama_stack_client 0.2.18
llama-stack-provider-lmeval 0.2.1
llama-stack-provider-trustyai-fms 0.2.0
The ERROR:
INFO 2025-08-26 11:19:52,529 __main__:185 server: Generator cancelled
+ Exception Group Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 77, in collapse_excgroups
| yield
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 271, in __call__
| async with anyio.create_task_group() as task_group:
| File "/opt/app-root/lib64/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
| await self.app(scope, receive, _send)
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 331, in __call__
| return await self.app(scope, receive, send_with_trace_id)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 373, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
| await route.handle(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
| await self.app(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
| await response(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
| with collapse_excgroups():
| File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
| self.gen.throw(value)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
| await func()
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
| async for chunk in self.body_iterator:
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/utils/context.py", line 29, in wrapper
| item = await gen.__anext__()
| ^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 187, in sse_generator
| await event_gen.aclose()
| ^^^^^^^^^^^^^^^^
| AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
+------------------------------------
Yes, the error is reproduced in the 0.2.18 llamastack version. I tested the latest tag from the downstream repository that include these versions:
llama_api_client 0.3.0 2
llama_stack 0.2.18 2
llama_stack_client 0.2.20 2
llama-stack-provider-lmeval 0.2.3 2
llama-stack-provider-trustyai-fms 0.2.1 2
The ERROR is happening in the end of the guidellm job:
INFO 2025-09-08 08:23:48,481 console_span_processor:48 telemetry: output: <openai.AsyncStream object at 0x7f27d49eeae0>
INFO 2025-09-08 08:23:48,493 console_span_processor:39 telemetry: 08:23:48.483 [END] InferenceRouter.openai_completion [StatusCode.OK] (50.73ms)
INFO 2025-09-08 08:23:48,494 console_span_processor:48 telemetry: output: <openai.AsyncStream object at 0x7f27d49eeae0>
INFO 2025-09-08 08:23:49,260 __main__:190 server: Generator cancelled
+ Exception Group Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
| yield
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 271, in __call__
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
| await self.app(scope, receive, _send)
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 340, in __call__
| return await self.app(scope, receive, send_with_trace_id)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 382, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
| await route.handle(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
| await self.app(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
| await response(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
| with collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
| self.gen.throw(value)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
| await func()
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
| async for chunk in self.body_iterator:
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/utils/context.py", line 29, in wrapper
| item = await gen.__anext__()
| ^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 192, in sse_generator
| await event_gen.aclose()
| ^^^^^^^^^^^^^^^^
| AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 340, in __call__
return await self.app(scope, receive, send_with_trace_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 382, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
await route.handle(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
await self.app(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
await response(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
with collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
await func()
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
async for chunk in self.body_iterator:
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/utils/context.py", line 29, in wrapper
item = await gen.__anext__()
^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 192, in sse_generator
await event_gen.aclose()
^^^^^^^^^^^^^^^^
AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
ERROR 2025-09-08 08:23:49,268 uvicorn.error:408 uncategorized: Exception in ASGI application
╭──────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮
│ /opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py:79 in collapse_excgroups │
│ │
│ 76 @contextmanager │
│ 77 def collapse_excgroups() -> Generator[None, None, None]: │
│ 78 │ try: │
│ ❱ 79 │ │ yield │
│ 80 │ except BaseException as exc: │
│ 81 │ │ if has_exceptiongroups: # pragma: no cover │
│ 82 │ │ │ while isinstance(exc, BaseExceptionGroup) and len(exc.exceptions) == 1: │
│ │
│ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:271 in __call__ │
│ │
│ 268 │ │ │ │ raise ClientDisconnect() │
│ 269 │ │ else: │
│ 270 │ │ │ with collapse_excgroups(): │
│ ❱ 271 │ │ │ │ async with anyio.create_task_group() as task_group: │
│ 272 │ │ │ │ │ │
│ 273 │ │ │ │ │ async def wrap(func: Callable[[], Awaitable[None]]) -> None: │
│ 274 │ │ │ │ │ │ await func() │
│ │
│ /opt/app-root/lib64/python3.12/site-packages/anyio/_backends/_asyncio.py:772 in __aexit__ │
│ │
│ 769 │ │ │ │ │ # added to self._exceptions so it's ok to break exception │
│ 770 │ │ │ │ │ # chaining and avoid adding a "During handling of above..." │
│ 771 │ │ │ │ │ # for each nesting level. │
│ ❱ 772 │ │ │ │ │ raise BaseExceptionGroup( │
│ 773 │ │ │ │ │ │ "unhandled errors in a TaskGroup", self._exceptions │
│ 774 │ │ │ │ │ ) from None │
│ 775 │ │ │ │ elif exc_val: │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
╭──────────────────────────────────────────────────────────── Sub-exception #1 ─────────────────────────────────────────────────────────────╮
│ ╭────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────╮ │
│ │ /opt/app-root/lib64/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py:403 in run_asgi │ │
│ │ │ │
│ │ 400 │ # ASGI exception wrapper │ │
│ │ 401 │ async def run_asgi(self, app: ASGI3Application) -> None: │ │
│ │ 402 │ │ try: │ │
│ │ ❱ 403 │ │ │ result = await app( # type: ignore[func-returns-value] │ │
│ │ 404 │ │ │ │ self.scope, self.receive, self.send │ │
│ │ 405 │ │ │ ) │ │
│ │ 406 │ │ except BaseException as exc: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/uvicorn/middleware/proxy_headers.py:60 in __call__ │ │
│ │ │ │
│ │ 57 │ │ │ │ │ port = 0 │ │
│ │ 58 │ │ │ │ │ scope["client"] = (host, port) │ │
│ │ 59 │ │ │ │
│ │ ❱ 60 │ │ return await self.app(scope, receive, send) │ │
│ │ 61 │ │
│ │ 62 │ │
│ │ 63 def _parse_raw_hosts(value: str) -> list[str]: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/fastapi/applications.py:1054 in __call__ │ │
│ │ │ │
│ │ 1051 │ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: │ │
│ │ 1052 │ │ if self.root_path: │ │
│ │ 1053 │ │ │ scope["root_path"] = self.root_path │ │
│ │ ❱ 1054 │ │ await super().__call__(scope, receive, send) │ │
│ │ 1055 │ │ │
│ │ 1056 │ def add_api_route( │ │
│ │ 1057 │ │ self, │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/applications.py:113 in __call__ │ │
│ │ │ │
│ │ 110 │ │ scope["app"] = self │ │
│ │ 111 │ │ if self.middleware_stack is None: │ │
│ │ 112 │ │ │ self.middleware_stack = self.build_middleware_stack() │ │
│ │ ❱ 113 │ │ await self.middleware_stack(scope, receive, send) │ │
│ │ 114 │ │ │
│ │ 115 │ def on_event(self, event_type: str) -> Callable: # type: ignore[type-arg] │ │
│ │ 116 │ │ return self.router.on_event(event_type) # pragma: no cover │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py:186 in __call__ │ │
│ │ │ │
│ │ 183 │ │ │ # We always continue to raise the exception. │ │
│ │ 184 │ │ │ # This allows servers to log the error, or allows test clients │ │
│ │ 185 │ │ │ # to optionally raise the error within the test case. │ │
│ │ ❱ 186 │ │ │ raise exc │ │
│ │ 187 │ │ │
│ │ 188 │ def format_line(self, index: int, line: str, frame_lineno: int, frame_index: int) -> │ │
│ │ str: │ │
│ │ 189 │ │ values = { │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py:164 in __call__ │ │
│ │ │ │
│ │ 161 │ │ │ await send(message) │ │
│ │ 162 │ │ │ │
│ │ 163 │ │ try: │ │
│ │ ❱ 164 │ │ │ await self.app(scope, receive, _send) │ │
│ │ 165 │ │ except Exception as exc: │ │
│ │ 166 │ │ │ request = Request(scope) │ │
│ │ 167 │ │ │ if self.debug: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py:340 in __call__ │ │
│ │ │ │
│ │ 337 │ │ │ await send(message) │ │
│ │ 338 │ │ │ │
│ │ 339 │ │ try: │ │
│ │ ❱ 340 │ │ │ return await self.app(scope, receive, send_with_trace_id) │ │
│ │ 341 │ │ finally: │ │
│ │ 342 │ │ │ await end_trace() │ │
│ │ 343 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py:382 in __call__ │ │
│ │ │ │
│ │ 379 │ │ │ │ │ # If version parsing fails, let the request through │ │
│ │ 380 │ │ │ │ │ pass │ │
│ │ 381 │ │ │ │
│ │ ❱ 382 │ │ return await self.app(scope, receive, send) │ │
│ │ 383 │ │
│ │ 384 │ │
│ │ 385 def main(args: argparse.Namespace | None = None): │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py:63 in __call__ │ │
│ │ │ │
│ │ 60 │ │ else: │ │
│ │ 61 │ │ │ conn = WebSocket(scope, receive, send) │ │
│ │ 62 │ │ │ │
│ │ ❱ 63 │ │ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) │ │
│ │ 64 │ │ │
│ │ 65 │ async def http_exception(self, request: Request, exc: Exception) -> Response: │ │
│ │ 66 │ │ assert isinstance(exc, HTTPException) │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:53 in wrapped_app │ │
│ │ │ │
│ │ 50 │ │ │ │ handler = _lookup_exception_handler(exception_handlers, exc) │ │
│ │ 51 │ │ │ │ │
│ │ 52 │ │ │ if handler is None: │ │
│ │ ❱ 53 │ │ │ │ raise exc │ │
│ │ 54 │ │ │ │ │
│ │ 55 │ │ │ if response_started: │ │
│ │ 56 │ │ │ │ raise RuntimeError("Caught handled exception, but response already │ │
│ │ started.") from exc │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:42 in wrapped_app │ │
│ │ │ │
│ │ 39 │ │ │ await send(message) │ │
│ │ 40 │ │ │ │
│ │ 41 │ │ try: │ │
│ │ ❱ 42 │ │ │ await app(scope, receive, sender) │ │
│ │ 43 │ │ except Exception as exc: │ │
│ │ 44 │ │ │ handler = None │ │
│ │ 45 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:716 in __call__ │ │
│ │ │ │
│ │ 713 │ │ """ │ │
│ │ 714 │ │ The main entry point to the Router class. │ │
│ │ 715 │ │ """ │ │
│ │ ❱ 716 │ │ await self.middleware_stack(scope, receive, send) │ │
│ │ 717 │ │ │
│ │ 718 │ async def app(self, scope: Scope, receive: Receive, send: Send) -> None: │ │
│ │ 719 │ │ assert scope["type"] in ("http", "websocket", "lifespan") │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:736 in app │ │
│ │ │ │
│ │ 733 │ │ │ match, child_scope = route.matches(scope) │ │
│ │ 734 │ │ │ if match == Match.FULL: │ │
│ │ 735 │ │ │ │ scope.update(child_scope) │ │
│ │ ❱ 736 │ │ │ │ await route.handle(scope, receive, send) │ │
│ │ 737 │ │ │ │ return │ │
│ │ 738 │ │ │ elif match == Match.PARTIAL and partial is None: │ │
│ │ 739 │ │ │ │ partial = route │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:290 in handle │ │
│ │ │ │
│ │ 287 │ │ │ │ response = PlainTextResponse("Method Not Allowed", status_code=405, │ │
│ │ headers=headers) │ │
│ │ 288 │ │ │ await response(scope, receive, send) │ │
│ │ 289 │ │ else: │ │
│ │ ❱ 290 │ │ │ await self.app(scope, receive, send) │ │
│ │ 291 │ │ │
│ │ 292 │ def __eq__(self, other: Any) -> bool: │ │
│ │ 293 │ │ return ( │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:78 in app │ │
│ │ │ │
│ │ 75 │ │ │ response = await f(request) │ │
│ │ 76 │ │ │ await response(scope, receive, send) │ │
│ │ 77 │ │ │ │
│ │ ❱ 78 │ │ await wrap_app_handling_exceptions(app, request)(scope, receive, send) │ │
│ │ 79 │ │ │
│ │ 80 │ return app │ │
│ │ 81 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:53 in wrapped_app │ │
│ │ │ │
│ │ 50 │ │ │ │ handler = _lookup_exception_handler(exception_handlers, exc) │ │
│ │ 51 │ │ │ │ │
│ │ 52 │ │ │ if handler is None: │ │
│ │ ❱ 53 │ │ │ │ raise exc │ │
│ │ 54 │ │ │ │ │
│ │ 55 │ │ │ if response_started: │ │
│ │ 56 │ │ │ │ raise RuntimeError("Caught handled exception, but response already │ │
│ │ started.") from exc │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:42 in wrapped_app │ │
│ │ │ │
│ │ 39 │ │ │ await send(message) │ │
│ │ 40 │ │ │ │
│ │ 41 │ │ try: │ │
│ │ ❱ 42 │ │ │ await app(scope, receive, sender) │ │
│ │ 43 │ │ except Exception as exc: │ │
│ │ 44 │ │ │ handler = None │ │
│ │ 45 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:76 in app │ │
│ │ │ │
│ │ 73 │ │ │ │
│ │ 74 │ │ async def app(scope: Scope, receive: Receive, send: Send) -> None: │ │
│ │ 75 │ │ │ response = await f(request) │ │
│ │ ❱ 76 │ │ │ await response(scope, receive, send) │ │
│ │ 77 │ │ │ │
│ │ 78 │ │ await wrap_app_handling_exceptions(app, request)(scope, receive, send) │ │
│ │ 79 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:270 in __call__ │ │
│ │ │ │
│ │ 267 │ │ │ except OSError: │ │
│ │ 268 │ │ │ │ raise ClientDisconnect() │ │
│ │ 269 │ │ else: │ │
│ │ ❱ 270 │ │ │ with collapse_excgroups(): │ │
│ │ 271 │ │ │ │ async with anyio.create_task_group() as task_group: │ │
│ │ 272 │ │ │ │ │ │ │
│ │ 273 │ │ │ │ │ async def wrap(func: Callable[[], Awaitable[None]]) -> None: │ │
│ │ │ │
│ │ /usr/lib64/python3.12/contextlib.py:158 in __exit__ │ │
│ │ │ │
│ │ 155 │ │ │ │ # tell if we get the same exception back │ │
│ │ 156 │ │ │ │ value = typ() │ │
│ │ 157 │ │ │ try: │ │
│ │ ❱ 158 │ │ │ │ self.gen.throw(value) │ │
│ │ 159 │ │ │ except StopIteration as exc: │ │
│ │ 160 │ │ │ │ # Suppress StopIteration *unless* it's the same exception that │ │
│ │ 161 │ │ │ │ # was passed to throw(). This prevents a StopIteration │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py:85 in collapse_excgroups │ │
│ │ │ │
│ │ 82 │ │ │ while isinstance(exc, BaseExceptionGroup) and len(exc.exceptions) == 1: │ │
│ │ 83 │ │ │ │ exc = exc.exceptions[0] │ │
│ │ 84 │ │ │ │
│ │ ❱ 85 │ │ raise exc │ │
│ │ 86 │ │
│ │ 87 │ │
│ │ 88 def get_route_path(scope: Scope) -> str: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:274 in wrap │ │
│ │ │ │
│ │ 271 │ │ │ │ async with anyio.create_task_group() as task_group: │ │
│ │ 272 │ │ │ │ │ │ │
│ │ 273 │ │ │ │ │ async def wrap(func: Callable[[], Awaitable[None]]) -> None: │ │
│ │ ❱ 274 │ │ │ │ │ │ await func() │ │
│ │ 275 │ │ │ │ │ │ task_group.cancel_scope.cancel() │ │
│ │ 276 │ │ │ │ │ │ │
│ │ 277 │ │ │ │ │ task_group.start_soon(wrap, partial(self.stream_response, send)) │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:254 in stream_response │ │
│ │ │ │
│ │ 251 │ │ │ │ "headers": self.raw_headers, │ │
│ │ 252 │ │ │ } │ │
│ │ 253 │ │ ) │ │
│ │ ❱ 254 │ │ async for chunk in self.body_iterator: │ │
│ │ 255 │ │ │ if not isinstance(chunk, (bytes, memoryview)): │ │
│ │ 256 │ │ │ │ chunk = chunk.encode(self.charset) │ │
│ │ 257 │ │ │ await send({"type": "http.response.body", "body": chunk, "more_body": True}) │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/utils/context.py:29 in wrapper │ │
│ │ │ │
│ │ 26 │ │ │ │ for context_var in context_vars: │ │
│ │ 27 │ │ │ │ │ context_var.set(initial_context_values[context_var.name]) │ │
│ │ 28 │ │ │ │ │ │
│ │ ❱ 29 │ │ │ │ item = await gen.__anext__() │ │
│ │ 30 │ │ │ │ │ │
│ │ 31 │ │ │ │ # Update our tracked values with any changes made during this iteration │ │
│ │ 32 │ │ │ │ for context_var in context_vars: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py:192 in sse_generator │ │
│ │ │ │
│ │ 189 │ except asyncio.CancelledError: │ │
│ │ 190 │ │ logger.info("Generator cancelled") │ │
│ │ 191 │ │ if event_gen: │ │
│ │ ❱ 192 │ │ │ await event_gen.aclose() │ │
│ │ 193 │ except Exception as e: │ │
│ │ 194 │ │ logger.exception("Error in sse_generator") │ │
│ │ 195 │ │ yield create_sse_event( │ │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ AttributeError: 'AsyncStream' object has no attribute 'aclose' │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The error is reproduced in the 0.2.20 llama-stack version. I tested the latest tag from the downstream repository that include these versions:
llama_api_client 0.3.0
llama_stack 0.2.20
llama_stack_client 0.2.20
llama-stack-provider-lmeval 0.2.3
llama-stack-provider-trustyai-fms 0.2.1
The ERROR is happening in the end of the guidellm job:
INFO 2025-09-09 13:59:47,411 __main__:192 core::server: Generator cancelled
INFO 2025-09-09 13:59:47,412 __main__:192 core::server: Generator cancelled
INFO 2025-09-09 13:59:47,413 __main__:192 core::server: Generator cancelled
INFO 2025-09-09 13:59:47,415 console_span_processor:39 telemetry: 13:59:47.404 [END] ModelsRoutingTable.get_provider_impl [StatusCode.OK] (0.66ms)
INFO 2025-09-09 13:59:47,416 console_span_processor:48 telemetry: output:
<llama_stack.providers.remote.inference.vllm.vllm.VLLMInferenceAdapter object at 0x7fd128d18e30>
+ Exception Group Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
| yield
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 271, in __call__
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
| await self.app(scope, receive, _send)
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 342, in __call__
| return await self.app(scope, receive, send_with_trace_id)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 384, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
| await route.handle(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
| await self.app(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
| await response(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
| with collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
| self.gen.throw(value)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
| await func()
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
| async for chunk in self.body_iterator:
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/utils/context.py", line 29, in wrapper
| item = await gen.__anext__()
| ^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 194, in sse_generator
| await event_gen.aclose()
| ^^^^^^^^^^^^^^^^
| AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 342, in __call__
return await self.app(scope, receive, send_with_trace_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 384, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
await route.handle(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
await self.app(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
await response(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
with collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
await func()
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
async for chunk in self.body_iterator:
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/utils/context.py", line 29, in wrapper
item = await gen.__anext__()
^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 194, in sse_generator
await event_gen.aclose()
^^^^^^^^^^^^^^^^
AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
ERROR 2025-09-09 13:59:47,423 uvicorn.error:408 uncategorized: Exception in ASGI application
╭──────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮
│ /opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py:79 in collapse_excgroups │
│ │
│ 76 @contextmanager │
│ 77 def collapse_excgroups() -> Generator[None, None, None]: │
│ 78 │ try: │
│ ❱ 79 │ │ yield │
│ 80 │ except BaseException as exc: │
│ 81 │ │ if has_exceptiongroups: # pragma: no cover │
│ 82 │ │ │ while isinstance(exc, BaseExceptionGroup) and len(exc.exceptions) == 1: │
│ │
│ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:271 in __call__ │
│ │
│ 268 │ │ │ │ raise ClientDisconnect() │
│ 269 │ │ else: │
│ 270 │ │ │ with collapse_excgroups(): │
│ ❱ 271 │ │ │ │ async with anyio.create_task_group() as task_group: │
│ 272 │ │ │ │ │ │
│ 273 │ │ │ │ │ async def wrap(func: Callable[[], Awaitable[None]]) -> None: │
│ 274 │ │ │ │ │ │ await func() │
│ │
│ /opt/app-root/lib64/python3.12/site-packages/anyio/_backends/_asyncio.py:772 in __aexit__ │
│ │
│ 769 │ │ │ │ │ # added to self._exceptions so it's ok to break exception │
│ 770 │ │ │ │ │ # chaining and avoid adding a "During handling of above..." │
│ 771 │ │ │ │ │ # for each nesting level. │
│ ❱ 772 │ │ │ │ │ raise BaseExceptionGroup( │
│ 773 │ │ │ │ │ │ "unhandled errors in a TaskGroup", self._exceptions │
│ 774 │ │ │ │ │ ) from None │
│ 775 │ │ │ │ elif exc_val: │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
╭──────────────────────────────────────────────────────────── Sub-exception #1 ─────────────────────────────────────────────────────────────╮
│ ╭────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────╮ │
│ │ /opt/app-root/lib64/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py:403 in run_asgi │ │
│ │ │ │
│ │ 400 │ # ASGI exception wrapper │ │
│ │ 401 │ async def run_asgi(self, app: ASGI3Application) -> None: │ │
│ │ 402 │ │ try: │ │
│ │ ❱ 403 │ │ │ result = await app( # type: ignore[func-returns-value] │ │
│ │ 404 │ │ │ │ self.scope, self.receive, self.send │ │
│ │ 405 │ │ │ ) │ │
│ │ 406 │ │ except BaseException as exc: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/uvicorn/middleware/proxy_headers.py:60 in __call__ │ │
│ │ │ │
│ │ 57 │ │ │ │ │ port = 0 │ │
│ │ 58 │ │ │ │ │ scope["client"] = (host, port) │ │
│ │ 59 │ │ │ │
│ │ ❱ 60 │ │ return await self.app(scope, receive, send) │ │
│ │ 61 │ │
│ │ 62 │ │
│ │ 63 def _parse_raw_hosts(value: str) -> list[str]: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/fastapi/applications.py:1054 in __call__ │ │
│ │ │ │
│ │ 1051 │ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: │ │
│ │ 1052 │ │ if self.root_path: │ │
│ │ 1053 │ │ │ scope["root_path"] = self.root_path │ │
│ │ ❱ 1054 │ │ await super().__call__(scope, receive, send) │ │
│ │ 1055 │ │ │
│ │ 1056 │ def add_api_route( │ │
│ │ 1057 │ │ self, │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/applications.py:113 in __call__ │ │
│ │ │ │
│ │ 110 │ │ scope["app"] = self │ │
│ │ 111 │ │ if self.middleware_stack is None: │ │
│ │ 112 │ │ │ self.middleware_stack = self.build_middleware_stack() │ │
│ │ ❱ 113 │ │ await self.middleware_stack(scope, receive, send) │ │
│ │ 114 │ │ │
│ │ 115 │ def on_event(self, event_type: str) -> Callable: # type: ignore[type-arg] │ │
│ │ 116 │ │ return self.router.on_event(event_type) # pragma: no cover │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py:186 in __call__ │ │
│ │ │ │
│ │ 183 │ │ │ # We always continue to raise the exception. │ │
│ │ 184 │ │ │ # This allows servers to log the error, or allows test clients │ │
│ │ 185 │ │ │ # to optionally raise the error within the test case. │ │
│ │ ❱ 186 │ │ │ raise exc │ │
│ │ 187 │ │ │
│ │ 188 │ def format_line(self, index: int, line: str, frame_lineno: int, frame_index: int) -> │ │
│ │ str: │ │
│ │ 189 │ │ values = { │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py:164 in __call__ │ │
│ │ │ │
│ │ 161 │ │ │ await send(message) │ │
│ │ 162 │ │ │ │
│ │ 163 │ │ try: │ │
│ │ ❱ 164 │ │ │ await self.app(scope, receive, _send) │ │
│ │ 165 │ │ except Exception as exc: │ │
│ │ 166 │ │ │ request = Request(scope) │ │
│ │ 167 │ │ │ if self.debug: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py:342 in __call__ │ │
│ │ │ │
│ │ 339 │ │ │ await send(message) │ │
│ │ 340 │ │ │ │
│ │ 341 │ │ try: │ │
│ │ ❱ 342 │ │ │ return await self.app(scope, receive, send_with_trace_id) │ │
│ │ 343 │ │ finally: │ │
│ │ 344 │ │ │ await end_trace() │ │
│ │ 345 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py:384 in __call__ │ │
│ │ │ │
│ │ 381 │ │ │ │ │ # If version parsing fails, let the request through │ │
│ │ 382 │ │ │ │ │ pass │ │
│ │ 383 │ │ │ │
│ │ ❱ 384 │ │ return await self.app(scope, receive, send) │ │
│ │ 385 │ │
│ │ 386 │ │
│ │ 387 def main(args: argparse.Namespace | None = None): │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py:63 in __call__ │ │
│ │ │ │
│ │ 60 │ │ else: │ │
│ │ 61 │ │ │ conn = WebSocket(scope, receive, send) │ │
│ │ 62 │ │ │ │
│ │ ❱ 63 │ │ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) │ │
│ │ 64 │ │ │
│ │ 65 │ async def http_exception(self, request: Request, exc: Exception) -> Response: │ │
│ │ 66 │ │ assert isinstance(exc, HTTPException) │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:53 in wrapped_app │ │
│ │ │ │
│ │ 50 │ │ │ │ handler = _lookup_exception_handler(exception_handlers, exc) │ │
│ │ 51 │ │ │ │ │
│ │ 52 │ │ │ if handler is None: │ │
│ │ ❱ 53 │ │ │ │ raise exc │ │
│ │ 54 │ │ │ │ │
│ │ 55 │ │ │ if response_started: │ │
│ │ 56 │ │ │ │ raise RuntimeError("Caught handled exception, but response already │ │
│ │ started.") from exc │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:42 in wrapped_app │ │
│ │ │ │
│ │ 39 │ │ │ await send(message) │ │
│ │ 40 │ │ │ │
│ │ 41 │ │ try: │ │
│ │ ❱ 42 │ │ │ await app(scope, receive, sender) │ │
│ │ 43 │ │ except Exception as exc: │ │
│ │ 44 │ │ │ handler = None │ │
│ │ 45 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:716 in __call__ │ │
│ │ │ │
│ │ 713 │ │ """ │ │
│ │ 714 │ │ The main entry point to the Router class. │ │
│ │ 715 │ │ """ │ │
│ │ ❱ 716 │ │ await self.middleware_stack(scope, receive, send) │ │
│ │ 717 │ │ │
│ │ 718 │ async def app(self, scope: Scope, receive: Receive, send: Send) -> None: │ │
│ │ 719 │ │ assert scope["type"] in ("http", "websocket", "lifespan") │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:736 in app │ │
│ │ │ │
│ │ 733 │ │ │ match, child_scope = route.matches(scope) │ │
│ │ 734 │ │ │ if match == Match.FULL: │ │
│ │ 735 │ │ │ │ scope.update(child_scope) │ │
│ │ ❱ 736 │ │ │ │ await route.handle(scope, receive, send) │ │
│ │ 737 │ │ │ │ return │ │
│ │ 738 │ │ │ elif match == Match.PARTIAL and partial is None: │ │
│ │ 739 │ │ │ │ partial = route │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:290 in handle │ │
│ │ │ │
│ │ 287 │ │ │ │ response = PlainTextResponse("Method Not Allowed", status_code=405, │ │
│ │ headers=headers) │ │
│ │ 288 │ │ │ await response(scope, receive, send) │ │
│ │ 289 │ │ else: │ │
│ │ ❱ 290 │ │ │ await self.app(scope, receive, send) │ │
│ │ 291 │ │ │
│ │ 292 │ def __eq__(self, other: Any) -> bool: │ │
│ │ 293 │ │ return ( │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:78 in app │ │
│ │ │ │
│ │ 75 │ │ │ response = await f(request) │ │
│ │ 76 │ │ │ await response(scope, receive, send) │ │
│ │ 77 │ │ │ │
│ │ ❱ 78 │ │ await wrap_app_handling_exceptions(app, request)(scope, receive, send) │ │
│ │ 79 │ │ │
│ │ 80 │ return app │ │
│ │ 81 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:53 in wrapped_app │ │
│ │ │ │
│ │ 50 │ │ │ │ handler = _lookup_exception_handler(exception_handlers, exc) │ │
│ │ 51 │ │ │ │ │
│ │ 52 │ │ │ if handler is None: │ │
│ │ ❱ 53 │ │ │ │ raise exc │ │
│ │ 54 │ │ │ │ │
│ │ 55 │ │ │ if response_started: │ │
│ │ 56 │ │ │ │ raise RuntimeError("Caught handled exception, but response already │ │
│ │ started.") from exc │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:42 in wrapped_app │ │
│ │ │ │
│ │ 39 │ │ │ await send(message) │ │
│ │ 40 │ │ │ │
│ │ 41 │ │ try: │ │
│ │ ❱ 42 │ │ │ await app(scope, receive, sender) │ │
│ │ 43 │ │ except Exception as exc: │ │
│ │ 44 │ │ │ handler = None │ │
│ │ 45 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:76 in app │ │
│ │ │ │
│ │ 73 │ │ │ │
│ │ 74 │ │ async def app(scope: Scope, receive: Receive, send: Send) -> None: │ │
│ │ 75 │ │ │ response = await f(request) │ │
│ │ ❱ 76 │ │ │ await response(scope, receive, send) │ │
│ │ 77 │ │ │ │
│ │ 78 │ │ await wrap_app_handling_exceptions(app, request)(scope, receive, send) │ │
│ │ 79 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:270 in __call__ │ │
│ │ │ │
│ │ 267 │ │ │ except OSError: │ │
│ │ 268 │ │ │ │ raise ClientDisconnect() │ │
│ │ 269 │ │ else: │ │
│ │ ❱ 270 │ │ │ with collapse_excgroups(): │ │
│ │ 271 │ │ │ │ async with anyio.create_task_group() as task_group: │ │
│ │ 272 │ │ │ │ │ │ │
│ │ 273 │ │ │ │ │ async def wrap(func: Callable[[], Awaitable[None]]) -> None: │ │
│ │ │ │
│ │ /usr/lib64/python3.12/contextlib.py:158 in __exit__ │ │
│ │ │ │
│ │ 155 │ │ │ │ # tell if we get the same exception back │ │
│ │ 156 │ │ │ │ value = typ() │ │
│ │ 157 │ │ │ try: │ │
│ │ ❱ 158 │ │ │ │ self.gen.throw(value) │ │
│ │ 159 │ │ │ except StopIteration as exc: │ │
│ │ 160 │ │ │ │ # Suppress StopIteration *unless* it's the same exception that │ │
│ │ 161 │ │ │ │ # was passed to throw(). This prevents a StopIteration │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py:85 in collapse_excgroups │ │
│ │ │ │
│ │ 82 │ │ │ while isinstance(exc, BaseExceptionGroup) and len(exc.exceptions) == 1: │ │
│ │ 83 │ │ │ │ exc = exc.exceptions[0] │ │
│ │ 84 │ │ │ │
│ │ ❱ 85 │ │ raise exc │ │
│ │ 86 │ │
│ │ 87 │ │
│ │ 88 def get_route_path(scope: Scope) -> str: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:274 in wrap │ │
│ │ │ │
│ │ 271 │ │ │ │ async with anyio.create_task_group() as task_group: │ │
│ │ 272 │ │ │ │ │ │ │
│ │ 273 │ │ │ │ │ async def wrap(func: Callable[[], Awaitable[None]]) -> None: │ │
│ │ ❱ 274 │ │ │ │ │ │ await func() │ │
│ │ 275 │ │ │ │ │ │ task_group.cancel_scope.cancel() │ │
│ │ 276 │ │ │ │ │ │ │
│ │ 277 │ │ │ │ │ task_group.start_soon(wrap, partial(self.stream_response, send)) │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:254 in stream_response │ │
│ │ │ │
│ │ 251 │ │ │ │ "headers": self.raw_headers, │ │
│ │ 252 │ │ │ } │ │
│ │ 253 │ │ ) │ │
│ │ ❱ 254 │ │ async for chunk in self.body_iterator: │ │
│ │ 255 │ │ │ if not isinstance(chunk, (bytes, memoryview)): │ │
│ │ 256 │ │ │ │ chunk = chunk.encode(self.charset) │ │
│ │ 257 │ │ │ await send({"type": "http.response.body", "body": chunk, "more_body": True}) │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/utils/context.py:29 in wrapper │ │
│ │ │ │
│ │ 26 │ │ │ │ for context_var in context_vars: │ │
│ │ 27 │ │ │ │ │ context_var.set(initial_context_values[context_var.name]) │ │
│ │ 28 │ │ │ │ │ │
│ │ ❱ 29 │ │ │ │ item = await gen.__anext__() │ │
│ │ 30 │ │ │ │ │ │
│ │ 31 │ │ │ │ # Update our tracked values with any changes made during this iteration │ │
│ │ 32 │ │ │ │ for context_var in context_vars: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py:194 in sse_generator │ │
│ │ │ │
│ │ 191 │ except asyncio.CancelledError: │ │
│ │ 192 │ │ logger.info("Generator cancelled") │ │
│ │ 193 │ │ if event_gen: │ │
│ │ ❱ 194 │ │ │ await event_gen.aclose() │ │
│ │ 195 │ except Exception as e: │ │
│ │ 196 │ │ logger.exception("Error in sse_generator") │ │
│ │ 197 │ │ yield create_sse_event( │ │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ AttributeError: 'AsyncStream' object has no attribute 'aclose' │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
The error is reproduced in the 0.2.21 llama-stack version. I tested the latest tag from the downstream repository that include these versions:
llama_stack 0.2.21
llama_stack_client 0.2.21
llama-stack-provider-lmeval 0.2.3
llama-stack-provider-trustyai-fms 0.2.1
The ERROR is happening in the end of the guidellm job:
INFO 2025-09-14 13:37:07,234 __main__:194 core::server: Generator cancelled
INFO 2025-09-14 13:37:07,235 __main__:194 core::server: Generator cancelled
+ Exception Group Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
| yield
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 271, in __call__
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
| await self.app(scope, receive, _send)
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 344, in __call__
| return await self.app(scope, receive, send_with_trace_id)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 386, in __call__
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
| await self.middleware_stack(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
| await route.handle(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
| await self.app(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
| await response(scope, receive, send)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
| with collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
| self.gen.throw(value)
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
| raise exc
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
| await func()
| File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
| async for chunk in self.body_iterator:
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/utils/context.py", line 29, in wrapper
| item = await gen.__anext__()
| ^^^^^^^^^^^^^^^^^^^^^
| File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 196, in sse_generator
| await event_gen.aclose()
| ^^^^^^^^^^^^^^^^
| AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 344, in __call__
return await self.app(scope, receive, send_with_trace_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 386, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 716, in __call__
await self.middleware_stack(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 736, in app
await route.handle(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 290, in handle
await self.app(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 78, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/routing.py", line 76, in app
await response(scope, receive, send)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 270, in __call__
with collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.12/contextlib.py", line 158, in __exit__
self.gen.throw(value)
File "/opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
raise exc
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 274, in wrap
await func()
File "/opt/app-root/lib64/python3.12/site-packages/starlette/responses.py", line 254, in stream_response
async for chunk in self.body_iterator:
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/utils/context.py", line 29, in wrapper
item = await gen.__anext__()
^^^^^^^^^^^^^^^^^^^^^
File "/opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py", line 196, in sse_generator
await event_gen.aclose()
^^^^^^^^^^^^^^^^
AttributeError: 'AsyncStream' object has no attribute 'aclose'. Did you mean: 'close'?
ERROR 2025-09-14 13:37:07,241 uvicorn.error:408 uncategorized: Exception in ASGI application
╭──────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮
│ /opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py:79 in collapse_excgroups │
│ │
│ 76 @contextmanager │
│ 77 def collapse_excgroups() -> Generator[None, None, None]: │
│ 78 │ try: │
│ ❱ 79 │ │ yield │
│ 80 │ except BaseException as exc: │
│ 81 │ │ if has_exceptiongroups: # pragma: no cover │
│ 82 │ │ │ while isinstance(exc, BaseExceptionGroup) and len(exc.exceptions) == 1: │
│ │
│ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:271 in __call__ │
│ │
│ 268 │ │ │ │ raise ClientDisconnect() │
│ 269 │ │ else: │
│ 270 │ │ │ with collapse_excgroups(): │
│ ❱ 271 │ │ │ │ async with anyio.create_task_group() as task_group: │
│ 272 │ │ │ │ │ │
│ 273 │ │ │ │ │ async def wrap(func: Callable[[], Awaitable[None]]) -> None: │
│ 274 │ │ │ │ │ │ await func() │
│ │
│ /opt/app-root/lib64/python3.12/site-packages/anyio/_backends/_asyncio.py:772 in __aexit__ │
│ │
│ 769 │ │ │ │ │ # added to self._exceptions so it's ok to break exception │
│ 770 │ │ │ │ │ # chaining and avoid adding a "During handling of above..." │
│ 771 │ │ │ │ │ # for each nesting level. │
│ ❱ 772 │ │ │ │ │ raise BaseExceptionGroup( │
│ 773 │ │ │ │ │ │ "unhandled errors in a TaskGroup", self._exceptions │
│ 774 │ │ │ │ │ ) from None │
│ 775 │ │ │ │ elif exc_val: │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
╭──────────────────────────────────────────────────────────── Sub-exception #1 ─────────────────────────────────────────────────────────────╮
│ ╭────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────╮ │
│ │ /opt/app-root/lib64/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py:403 in run_asgi │ │
│ │ │ │
│ │ 400 │ # ASGI exception wrapper │ │
│ │ 401 │ async def run_asgi(self, app: ASGI3Application) -> None: │ │
│ │ 402 │ │ try: │ │
│ │ ❱ 403 │ │ │ result = await app( # type: ignore[func-returns-value] │ │
│ │ 404 │ │ │ │ self.scope, self.receive, self.send │ │
│ │ 405 │ │ │ ) │ │
│ │ 406 │ │ except BaseException as exc: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/uvicorn/middleware/proxy_headers.py:60 in __call__ │ │
│ │ │ │
│ │ 57 │ │ │ │ │ port = 0 │ │
│ │ 58 │ │ │ │ │ scope["client"] = (host, port) │ │
│ │ 59 │ │ │ │
│ │ ❱ 60 │ │ return await self.app(scope, receive, send) │ │
│ │ 61 │ │
│ │ 62 │ │
│ │ 63 def _parse_raw_hosts(value: str) -> list[str]: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/fastapi/applications.py:1054 in __call__ │ │
│ │ │ │
│ │ 1051 │ async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None: │ │
│ │ 1052 │ │ if self.root_path: │ │
│ │ 1053 │ │ │ scope["root_path"] = self.root_path │ │
│ │ ❱ 1054 │ │ await super().__call__(scope, receive, send) │ │
│ │ 1055 │ │ │
│ │ 1056 │ def add_api_route( │ │
│ │ 1057 │ │ self, │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/applications.py:113 in __call__ │ │
│ │ │ │
│ │ 110 │ │ scope["app"] = self │ │
│ │ 111 │ │ if self.middleware_stack is None: │ │
│ │ 112 │ │ │ self.middleware_stack = self.build_middleware_stack() │ │
│ │ ❱ 113 │ │ await self.middleware_stack(scope, receive, send) │ │
│ │ 114 │ │ │
│ │ 115 │ def on_event(self, event_type: str) -> Callable: # type: ignore[type-arg] │ │
│ │ 116 │ │ return self.router.on_event(event_type) # pragma: no cover │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py:186 in __call__ │ │
│ │ │ │
│ │ 183 │ │ │ # We always continue to raise the exception. │ │
│ │ 184 │ │ │ # This allows servers to log the error, or allows test clients │ │
│ │ 185 │ │ │ # to optionally raise the error within the test case. │ │
│ │ ❱ 186 │ │ │ raise exc │ │
│ │ 187 │ │ │
│ │ 188 │ def format_line(self, index: int, line: str, frame_lineno: int, frame_index: int) -> │ │
│ │ str: │ │
│ │ 189 │ │ values = { │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/middleware/errors.py:164 in __call__ │ │
│ │ │ │
│ │ 161 │ │ │ await send(message) │ │
│ │ 162 │ │ │ │
│ │ 163 │ │ try: │ │
│ │ ❱ 164 │ │ │ await self.app(scope, receive, _send) │ │
│ │ 165 │ │ except Exception as exc: │ │
│ │ 166 │ │ │ request = Request(scope) │ │
│ │ 167 │ │ │ if self.debug: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py:344 in __call__ │ │
│ │ │ │
│ │ 341 │ │ │ await send(message) │ │
│ │ 342 │ │ │ │
│ │ 343 │ │ try: │ │
│ │ ❱ 344 │ │ │ return await self.app(scope, receive, send_with_trace_id) │ │
│ │ 345 │ │ finally: │ │
│ │ 346 │ │ │ await end_trace() │ │
│ │ 347 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py:386 in __call__ │ │
│ │ │ │
│ │ 383 │ │ │ │ │ # If version parsing fails, let the request through │ │
│ │ 384 │ │ │ │ │ pass │ │
│ │ 385 │ │ │ │
│ │ ❱ 386 │ │ return await self.app(scope, receive, send) │ │
│ │ 387 │ │
│ │ 388 │ │
│ │ 389 def main(args: argparse.Namespace | None = None): │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/middleware/exceptions.py:63 in __call__ │ │
│ │ │ │
│ │ 60 │ │ else: │ │
│ │ 61 │ │ │ conn = WebSocket(scope, receive, send) │ │
│ │ 62 │ │ │ │
│ │ ❱ 63 │ │ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) │ │
│ │ 64 │ │ │
│ │ 65 │ async def http_exception(self, request: Request, exc: Exception) -> Response: │ │
│ │ 66 │ │ assert isinstance(exc, HTTPException) │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:53 in wrapped_app │ │
│ │ │ │
│ │ 50 │ │ │ │ handler = _lookup_exception_handler(exception_handlers, exc) │ │
│ │ 51 │ │ │ │ │
│ │ 52 │ │ │ if handler is None: │ │
│ │ ❱ 53 │ │ │ │ raise exc │ │
│ │ 54 │ │ │ │ │
│ │ 55 │ │ │ if response_started: │ │
│ │ 56 │ │ │ │ raise RuntimeError("Caught handled exception, but response already │ │
│ │ started.") from exc │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:42 in wrapped_app │ │
│ │ │ │
│ │ 39 │ │ │ await send(message) │ │
│ │ 40 │ │ │ │
│ │ 41 │ │ try: │ │
│ │ ❱ 42 │ │ │ await app(scope, receive, sender) │ │
│ │ 43 │ │ except Exception as exc: │ │
│ │ 44 │ │ │ handler = None │ │
│ │ 45 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:716 in __call__ │ │
│ │ │ │
│ │ 713 │ │ """ │ │
│ │ 714 │ │ The main entry point to the Router class. │ │
│ │ 715 │ │ """ │ │
│ │ ❱ 716 │ │ await self.middleware_stack(scope, receive, send) │ │
│ │ 717 │ │ │
│ │ 718 │ async def app(self, scope: Scope, receive: Receive, send: Send) -> None: │ │
│ │ 719 │ │ assert scope["type"] in ("http", "websocket", "lifespan") │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:736 in app │ │
│ │ │ │
│ │ 733 │ │ │ match, child_scope = route.matches(scope) │ │
│ │ 734 │ │ │ if match == Match.FULL: │ │
│ │ 735 │ │ │ │ scope.update(child_scope) │ │
│ │ ❱ 736 │ │ │ │ await route.handle(scope, receive, send) │ │
│ │ 737 │ │ │ │ return │ │
│ │ 738 │ │ │ elif match == Match.PARTIAL and partial is None: │ │
│ │ 739 │ │ │ │ partial = route │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:290 in handle │ │
│ │ │ │
│ │ 287 │ │ │ │ response = PlainTextResponse("Method Not Allowed", status_code=405, │ │
│ │ headers=headers) │ │
│ │ 288 │ │ │ await response(scope, receive, send) │ │
│ │ 289 │ │ else: │ │
│ │ ❱ 290 │ │ │ await self.app(scope, receive, send) │ │
│ │ 291 │ │ │
│ │ 292 │ def __eq__(self, other: Any) -> bool: │ │
│ │ 293 │ │ return ( │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:78 in app │ │
│ │ │ │
│ │ 75 │ │ │ response = await f(request) │ │
│ │ 76 │ │ │ await response(scope, receive, send) │ │
│ │ 77 │ │ │ │
│ │ ❱ 78 │ │ await wrap_app_handling_exceptions(app, request)(scope, receive, send) │ │
│ │ 79 │ │ │
│ │ 80 │ return app │ │
│ │ 81 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:53 in wrapped_app │ │
│ │ │ │
│ │ 50 │ │ │ │ handler = _lookup_exception_handler(exception_handlers, exc) │ │
│ │ 51 │ │ │ │ │
│ │ 52 │ │ │ if handler is None: │ │
│ │ ❱ 53 │ │ │ │ raise exc │ │
│ │ 54 │ │ │ │ │
│ │ 55 │ │ │ if response_started: │ │
│ │ 56 │ │ │ │ raise RuntimeError("Caught handled exception, but response already │ │
│ │ started.") from exc │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_exception_handler.py:42 in wrapped_app │ │
│ │ │ │
│ │ 39 │ │ │ await send(message) │ │
│ │ 40 │ │ │ │
│ │ 41 │ │ try: │ │
│ │ ❱ 42 │ │ │ await app(scope, receive, sender) │ │
│ │ 43 │ │ except Exception as exc: │ │
│ │ 44 │ │ │ handler = None │ │
│ │ 45 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/routing.py:76 in app │ │
│ │ │ │
│ │ 73 │ │ │ │
│ │ 74 │ │ async def app(scope: Scope, receive: Receive, send: Send) -> None: │ │
│ │ 75 │ │ │ response = await f(request) │ │
│ │ ❱ 76 │ │ │ await response(scope, receive, send) │ │
│ │ 77 │ │ │ │
│ │ 78 │ │ await wrap_app_handling_exceptions(app, request)(scope, receive, send) │ │
│ │ 79 │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:270 in __call__ │ │
│ │ │ │
│ │ 267 │ │ │ except OSError: │ │
│ │ 268 │ │ │ │ raise ClientDisconnect() │ │
│ │ 269 │ │ else: │ │
│ │ ❱ 270 │ │ │ with collapse_excgroups(): │ │
│ │ 271 │ │ │ │ async with anyio.create_task_group() as task_group: │ │
│ │ 272 │ │ │ │ │ │ │
│ │ 273 │ │ │ │ │ async def wrap(func: Callable[[], Awaitable[None]]) -> None: │ │
│ │ │ │
│ │ /usr/lib64/python3.12/contextlib.py:158 in __exit__ │ │
│ │ │ │
│ │ 155 │ │ │ │ # tell if we get the same exception back │ │
│ │ 156 │ │ │ │ value = typ() │ │
│ │ 157 │ │ │ try: │ │
│ │ ❱ 158 │ │ │ │ self.gen.throw(value) │ │
│ │ 159 │ │ │ except StopIteration as exc: │ │
│ │ 160 │ │ │ │ # Suppress StopIteration *unless* it's the same exception that │ │
│ │ 161 │ │ │ │ # was passed to throw(). This prevents a StopIteration │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/_utils.py:85 in collapse_excgroups │ │
│ │ │ │
│ │ 82 │ │ │ while isinstance(exc, BaseExceptionGroup) and len(exc.exceptions) == 1: │ │
│ │ 83 │ │ │ │ exc = exc.exceptions[0] │ │
│ │ 84 │ │ │ │
│ │ ❱ 85 │ │ raise exc │ │
│ │ 86 │ │
│ │ 87 │ │
│ │ 88 def get_route_path(scope: Scope) -> str: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:274 in wrap │ │
│ │ │ │
│ │ 271 │ │ │ │ async with anyio.create_task_group() as task_group: │ │
│ │ 272 │ │ │ │ │ │ │
│ │ 273 │ │ │ │ │ async def wrap(func: Callable[[], Awaitable[None]]) -> None: │ │
│ │ ❱ 274 │ │ │ │ │ │ await func() │ │
│ │ 275 │ │ │ │ │ │ task_group.cancel_scope.cancel() │ │
│ │ 276 │ │ │ │ │ │ │
│ │ 277 │ │ │ │ │ task_group.start_soon(wrap, partial(self.stream_response, send)) │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/starlette/responses.py:254 in stream_response │ │
│ │ │ │
│ │ 251 │ │ │ │ "headers": self.raw_headers, │ │
│ │ 252 │ │ │ } │ │
│ │ 253 │ │ ) │ │
│ │ ❱ 254 │ │ async for chunk in self.body_iterator: │ │
│ │ 255 │ │ │ if not isinstance(chunk, (bytes, memoryview)): │ │
│ │ 256 │ │ │ │ chunk = chunk.encode(self.charset) │ │
│ │ 257 │ │ │ await send({"type": "http.response.body", "body": chunk, "more_body": True}) │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/utils/context.py:29 in wrapper │ │
│ │ │ │
│ │ 26 │ │ │ │ for context_var in context_vars: │ │
│ │ 27 │ │ │ │ │ context_var.set(initial_context_values[context_var.name]) │ │
│ │ 28 │ │ │ │ │ │
│ │ ❱ 29 │ │ │ │ item = await gen.__anext__() │ │
│ │ 30 │ │ │ │ │ │
│ │ 31 │ │ │ │ # Update our tracked values with any changes made during this iteration │ │
│ │ 32 │ │ │ │ for context_var in context_vars: │ │
│ │ │ │
│ │ /opt/app-root/lib64/python3.12/site-packages/llama_stack/core/server/server.py:196 in sse_generator │ │
│ │ │ │
│ │ 193 │ except asyncio.CancelledError: │ │
│ │ 194 │ │ logger.info("Generator cancelled") │ │
│ │ 195 │ │ if event_gen: │ │
│ │ ❱ 196 │ │ │ await event_gen.aclose() │ │
│ │ 197 │ except Exception as e: │ │
│ │ 198 │ │ logger.exception("Error in sse_generator") │ │
│ │ 199 │ │ yield create_sse_event( │ │
│ ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ │
│ AttributeError: 'AsyncStream' object has no attribute 'aclose' │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
This issue has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@ashwinb this error is still reproduced, can I take it, it looks like an easy fix in server.py replacing await event_gen.aclose() (which doesn't exist for AsyncStream in the openai with event_gen.close() or better yet check if the object has that attribute as I notice its dynamically assigned function.
generally speaking this happens when streaming requests are cancelled (probably happens a lot with guidellm).
reproduction script based of ollama: https://gist.github.com/r-bit-rry/dea4f8fbb81c446f5db50ea7abd6379b
