Sam Stoelinga

vllm 0.6.6 now supports S3 compatible storage using RunAI model stream. That would allow setting all those parameters using environment variables: https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/env-vars.md I will test the new vLLM release and...

Thank you both! Amazing work. @liebman didn't you have to add this flag to vLLM args? ``` --load-format runai_streamer ``` I noticed in your example, you didn't set it: ```...

@jez4 the AWS_ENDPOINT_URL should already get propogated to the modelLoading pod: https://github.com/substratusai/kubeai/blob/d6e393ca76f11da76b3a6db74b737b94d1a4f057/internal/modelcontroller/cache.go#L320 What version are you running? @ffais and @liebman were you able to get loading from s3 compatible storage...

Thanks for the PR! I am happy to merge this as long as the setting API key is not required. I can do a more thorough review afterwards. Please also...

Infinity is OpenAI compatible. KubeAI has switched to infinity.

Is there a doc on how to set logging level? This was the only thing that came up when I searched for "Jax docs logging level" I am assuming what...

We can probably close this issue, it's already supported: https://github.com/search?q=repo%3Ajax-ml%2Fjax%20JAX_LOGGING_LEVEL&type=code This is how you do it: ``` export JAX_LOGGING_LEVEL="DEBUG" ```

Alternatively, you can also deploy KubeAI and open webui separately and take a look at the helm values we use by default for openwebui. Also will update openweb ui as...

Maybe the thing you were missing was `helm dependency update` after updating open webui chart version in Chart.yaml?

Yeah that was all I did and it worked fine. Tested it yesterday and today. On Sun, Apr 27, 2025, 8:32 PM ChengBinTham ***@***.***> wrote: > *cbtham* left a comment...