Sam Stoelinga
Sam Stoelinga
Error observed which triggered a restart: ``` 2024-04-25T00:55:18Z ERROR setup starting messenger {"error": "pubsub (code=InvalidArgument): rpc error: code = InvalidArgument desc = Some acknowledgement ids in the request were invalid....
Sometimes vLLM gets into a broken state and these errors will be shown for any request: ``` lark.exceptions.UnexpectedToken: Unexpected token Token('LBRACE', '{') at line 11, column 2. Expected one of:...
Troubleshooting issues with a faulty message becomes hard if all you have is a message id from e.g. pubsub. It would be nice to also log the metadata of each...
Any request that's pending for more than 30 minutes should return a 408 HTTP timout error This timeout should be configurable. A value of 0 would disable it such that...
``` 2024/03/29 06:01:03 Failed to aggregate stats: getting stats: 10.100.1.173:8080: decoding response body: invalid character 'B' looking for beginning of value 2024/03/29 06:01:03 Failed to aggregate stats: getting stats: 10.100.1.174:8080:...
Currently it seems Lingo is quickly creating more replicas as requests are incoming while the pod isn't ready to serve yet. It should be configurable how many requests a single...
@nstogner suggested to use: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/log which uses logr.
https://docs.vllm.ai/en/latest/serving/metrics.html