aibrix
aibrix copied to clipboard
vllm-project
Questions about deployment verification：{"error":{"code":500,"message":"invalid character 'u' looking for beginning of value"}}
aibrix version: 0.2.1 k8s version: v1.25.14
Deploy aibrix according to the official documentation and deploy the base model, but an error occurs when calling curl -v http://${ENDPOINT}/v1/models：
* Trying 10.0.102.94:1422...
* Connected to 10.0.102.94 (10.0.102.94) port 1422 (#0)
> GET /v1/models HTTP/1.1
> Host: 10.0.102.94:1422
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< x-error-response-unmarshal: true
< content-type:
< content-length: 87
< date: Fri, 09 May 2025 02:55:39 GMT
<
* Connection #0 to host 10.0.102.94 left intact
{"error":{"code":500,"message":"invalid character 'u' looking for beginning of value"}}
The establishment status is as follows：
envoy-aibrix-system-aibrix-eg-903790dc-fccbb99cd-4b9jg log:
{"start_time":"2025-05-09T02:52:58.493Z","method":"GET","x-envoy-origin-path":"/v1/models","protocol":"HTTP/1.1","response_code":"500","response_flags":"UPE","response_code_details":"","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"87","duration":"1","x-envoy-upstream-service-time":"-","x-forwarded-for":"10.244.64.64","user-agent":"curl/7.79.1","x-request-id":"eee2a978-fb05-4a45-aea8-0521daf10eea",":authority":"10.0.102.94:1422","upstream_host":"10.244.167.226:50052","upstream_cluster":"httproute/aibrix-system/aibrix-reserved-router/rule/0","upstream_local_address":"10.244.227.51:51240","downstream_local_address":"10.244.227.51:10080","downstream_remote_address":"10.244.64.64:10721","requested_server_name":"-","route_name":"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/0/*"}
{"start_time":"2025-05-09T02:55:39.952Z","method":"GET","x-envoy-origin-path":"/v1/models","protocol":"HTTP/1.1","response_code":"500","response_flags":"UPE","response_code_details":"","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"87","duration":"1","x-envoy-upstream-service-time":"-","x-forwarded-for":"10.244.64.64","user-agent":"curl/7.79.1","x-request-id":"09cea926-2f19-47df-9e53-e10f9e49d043",":authority":"10.0.102.94:1422","upstream_host":"10.244.167.226:50052","upstream_cluster":"httproute/aibrix-system/aibrix-reserved-router/rule/0","upstream_local_address":"10.244.227.51:54562","downstream_local_address":"10.244.227.51:10080","downstream_remote_address":"10.244.64.64:31778","requested_server_name":"-","route_name":"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/0/*"}
aibrix-gateway-plugins-6f76c4ccf7-bhg6g log:
I0509 02:45:52.377876 1 gateway.go:221] "-- In RequestHeaders processing ..." requestID="875ee759-819a-4d20-ba75-880e403b5ac1"
I0509 02:45:52.378490 1 gateway.go:393] "-- In ResponseHeaders processing ..." requestID="875ee759-819a-4d20-ba75-880e403b5ac1"
I0509 02:45:52.378631 1 gateway.go:426] "-- In ResponseBody processing ..." requestID="875ee759-819a-4d20-ba75-880e403b5ac1" endOfStream=true
E0509 02:45:52.378653 1 gateway.go:488] "error to unmarshal response" err="invalid character 'u' looking for beginning of value" requestID="875ee759-819a-4d20-ba75-880e403b5ac1" responseBody="upstream connect error or disconnect/reset before headers. reset reason: protocol error"
I0509 02:52:07.068626 1 gateway.go:220]
I0509 02:52:07.068664 1 gateway.go:221] "-- In RequestHeaders processing ..." requestID="e208c152-db4b-4a4a-a2bc-4a35332ffbf5"
I0509 02:52:07.069327 1 gateway.go:393] "-- In ResponseHeaders processing ..." requestID="e208c152-db4b-4a4a-a2bc-4a35332ffbf5"
I0509 02:52:07.069480 1 gateway.go:426] "-- In ResponseBody processing ..." requestID="e208c152-db4b-4a4a-a2bc-4a35332ffbf5" endOfStream=true
E0509 02:52:07.069505 1 gateway.go:488] "error to unmarshal response" err="invalid character 'u' looking for beginning of value" requestID="e208c152-db4b-4a4a-a2bc-4a35332ffbf5" responseBody="upstream connect error or disconnect/reset before headers. reset reason: protocol error"
I0509 02:52:58.492211 1 gateway.go:220]
I0509 02:52:58.492230 1 gateway.go:221] "-- In RequestHeaders processing ..." requestID="c81690b2-9adb-405b-a8af-0ae148accac1"
I0509 02:52:58.492713 1 gateway.go:393] "-- In ResponseHeaders processing ..." requestID="c81690b2-9adb-405b-a8af-0ae148accac1"
I0509 02:52:58.492888 1 gateway.go:426] "-- In ResponseBody processing ..." requestID="c81690b2-9adb-405b-a8af-0ae148accac1" endOfStream=true
E0509 02:52:58.492927 1 gateway.go:488] "error to unmarshal response" err="invalid character 'u' looking for beginning of value" requestID="c81690b2-9adb-405b-a8af-0ae148accac1" responseBody="upstream connect error or disconnect/reset before headers. reset reason: protocol error"
I0509 02:55:39.951694 1 gateway.go:220]
I0509 02:55:39.951712 1 gateway.go:221] "-- In RequestHeaders processing ..." requestID="0af76556-239b-4269-9a52-b7029e874c96"
I0509 02:55:39.952297 1 gateway.go:393] "-- In ResponseHeaders processing ..." requestID="0af76556-239b-4269-9a52-b7029e874c96"
I0509 02:55:39.952460 1 gateway.go:426] "-- In ResponseBody processing ..." requestID="0af76556-239b-4269-9a52-b7029e874c96" endOfStream=true
E0509 02:55:39.952483 1 gateway.go:488] "error to unmarshal response" err="invalid character 'u' looking for beginning of value" requestID="0af76556-239b-4269-9a52-b7029e874c96" responseBody="upstream connect error or disconnect/reset before headers. reset reason: protocol error"
gateway and httproute:
I have another question. Where can I modify the image of envoy-aibrix-system-aibrix-eg? I didn't find the relevant settings in the deployment file.
aibrix version: 0.2.1 k8s version: v1.25.14
Deploy aibrix according to the official documentation and deploy the base model, but an error occurs when calling curl -v http://${ENDPOINT}/v1/models：
* Trying 10.0.102.94:1422... * Connected to 10.0.102.94 (10.0.102.94) port 1422 (#0) > GET /v1/models HTTP/1.1 > Host: 10.0.102.94:1422 > User-Agent: curl/7.79.1 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 500 Internal Server Error < x-error-response-unmarshal: true < content-type: < content-length: 87 < date: Fri, 09 May 2025 02:55:39 GMT < * Connection #0 to host 10.0.102.94 left intact {"error":{"code":500,"message":"invalid character 'u' looking for beginning of value"}}The establishment status is as follows：
envoy-aibrix-system-aibrix-eg-903790dc-fccbb99cd-4b9jg log:
{"start_time":"2025-05-09T02:52:58.493Z","method":"GET","x-envoy-origin-path":"/v1/models","protocol":"HTTP/1.1","response_code":"500","response_flags":"UPE","response_code_details":"","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"87","duration":"1","x-envoy-upstream-service-time":"-","x-forwarded-for":"10.244.64.64","user-agent":"curl/7.79.1","x-request-id":"eee2a978-fb05-4a45-aea8-0521daf10eea",":authority":"10.0.102.94:1422","upstream_host":"10.244.167.226:50052","upstream_cluster":"httproute/aibrix-system/aibrix-reserved-router/rule/0","upstream_local_address":"10.244.227.51:51240","downstream_local_address":"10.244.227.51:10080","downstream_remote_address":"10.244.64.64:10721","requested_server_name":"-","route_name":"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/0/*"} {"start_time":"2025-05-09T02:55:39.952Z","method":"GET","x-envoy-origin-path":"/v1/models","protocol":"HTTP/1.1","response_code":"500","response_flags":"UPE","response_code_details":"","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"87","duration":"1","x-envoy-upstream-service-time":"-","x-forwarded-for":"10.244.64.64","user-agent":"curl/7.79.1","x-request-id":"09cea926-2f19-47df-9e53-e10f9e49d043",":authority":"10.0.102.94:1422","upstream_host":"10.244.167.226:50052","upstream_cluster":"httproute/aibrix-system/aibrix-reserved-router/rule/0","upstream_local_address":"10.244.227.51:54562","downstream_local_address":"10.244.227.51:10080","downstream_remote_address":"10.244.64.64:31778","requested_server_name":"-","route_name":"httproute/aibrix-system/aibrix-reserved-router/rule/0/match/0/*"}aibrix-gateway-plugins-6f76c4ccf7-bhg6g log:
I0509 02:45:52.377876 1 gateway.go:221] "-- In RequestHeaders processing ..." requestID="875ee759-819a-4d20-ba75-880e403b5ac1" I0509 02:45:52.378490 1 gateway.go:393] "-- In ResponseHeaders processing ..." requestID="875ee759-819a-4d20-ba75-880e403b5ac1" I0509 02:45:52.378631 1 gateway.go:426] "-- In ResponseBody processing ..." requestID="875ee759-819a-4d20-ba75-880e403b5ac1" endOfStream=true E0509 02:45:52.378653 1 gateway.go:488] "error to unmarshal response" err="invalid character 'u' looking for beginning of value" requestID="875ee759-819a-4d20-ba75-880e403b5ac1" responseBody="upstream connect error or disconnect/reset before headers. reset reason: protocol error" I0509 02:52:07.068626 1 gateway.go:220] I0509 02:52:07.068664 1 gateway.go:221] "-- In RequestHeaders processing ..." requestID="e208c152-db4b-4a4a-a2bc-4a35332ffbf5" I0509 02:52:07.069327 1 gateway.go:393] "-- In ResponseHeaders processing ..." requestID="e208c152-db4b-4a4a-a2bc-4a35332ffbf5" I0509 02:52:07.069480 1 gateway.go:426] "-- In ResponseBody processing ..." requestID="e208c152-db4b-4a4a-a2bc-4a35332ffbf5" endOfStream=true E0509 02:52:07.069505 1 gateway.go:488] "error to unmarshal response" err="invalid character 'u' looking for beginning of value" requestID="e208c152-db4b-4a4a-a2bc-4a35332ffbf5" responseBody="upstream connect error or disconnect/reset before headers. reset reason: protocol error" I0509 02:52:58.492211 1 gateway.go:220] I0509 02:52:58.492230 1 gateway.go:221] "-- In RequestHeaders processing ..." requestID="c81690b2-9adb-405b-a8af-0ae148accac1" I0509 02:52:58.492713 1 gateway.go:393] "-- In ResponseHeaders processing ..." requestID="c81690b2-9adb-405b-a8af-0ae148accac1" I0509 02:52:58.492888 1 gateway.go:426] "-- In ResponseBody processing ..." requestID="c81690b2-9adb-405b-a8af-0ae148accac1" endOfStream=true E0509 02:52:58.492927 1 gateway.go:488] "error to unmarshal response" err="invalid character 'u' looking for beginning of value" requestID="c81690b2-9adb-405b-a8af-0ae148accac1" responseBody="upstream connect error or disconnect/reset before headers. reset reason: protocol error" I0509 02:55:39.951694 1 gateway.go:220] I0509 02:55:39.951712 1 gateway.go:221] "-- In RequestHeaders processing ..." requestID="0af76556-239b-4269-9a52-b7029e874c96" I0509 02:55:39.952297 1 gateway.go:393] "-- In ResponseHeaders processing ..." requestID="0af76556-239b-4269-9a52-b7029e874c96" I0509 02:55:39.952460 1 gateway.go:426] "-- In ResponseBody processing ..." requestID="0af76556-239b-4269-9a52-b7029e874c96" endOfStream=true E0509 02:55:39.952483 1 gateway.go:488] "error to unmarshal response" err="invalid character 'u' looking for beginning of value" requestID="0af76556-239b-4269-9a52-b7029e874c96" responseBody="upstream connect error or disconnect/reset before headers. reset reason: protocol error"gateway and httproute:
I have another question. Where can I modify the image of envoy-aibrix-system-aibrix-eg? I didn't find the relevant settings in the deployment file.
I found that calling /v1/models will report {"error":{"code":500,"message":"invalid character 'u' looking for beginning of value"}}, but calling /v1/chat/completions will respond normally. There is another problem. When I set Rate Limiting in the request header, it will also report a 500 error, as follows:
[root@system-test-102-94 lgy]# curl -v 10.0.102.94:1422/v1/chat/completions -H "user: lgy" -H "Content-Type: application/json" -d '{
"model": "deepseek-r1-distill-llama-8b",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'
* Trying 10.0.102.94:1422...
* Connected to 10.0.102.94 (10.0.102.94) port 1422 (#0)
> POST /v1/chat/completions HTTP/1.1
> Host: 10.0.102.94:1422
> User-Agent: curl/7.79.1
> Accept: */*
> user: lgy
> Content-Type: application/json
> Content-Length: 141
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< x-error-user: true
< content-type:
< content-length: 45
< date: Fri, 09 May 2025 06:47:55 GMT
<
* Connection #0 to host 10.0.102.94 left intact
{"error":{"code":500,"message":"redis: nil"}}
- I am assuming you are using 0.2.1 release, and it does not have /v1/model implementation. It will be part of 0.3.0 (planned for this week).
- For rate limiting, did you create a user before? https://github.com/vllm-project/aibrix/blob/main/pkg/metadata/README.md
- Please share logs from
kubectl -n aibrix-system logs aibrix-gateway-plugins-6f....
To update envoy-proxy image, please update here. Since you are on 0.2.1 release, please delete default part of installation and re-create it.