Mattia Bradascio comments

Results 17 comments of


                                            Mattia Bradascio

Broken since docker 26

Disabling containerd does it on OSX (Docker Desktop).

What is the best way to multi model serving on local storage?

For this I'd recommend setting up local storage that is S3 compatible like Minio 👍🏼

Loading models from an S3 location instead of local path

I have another issue where the default runai streamer simply terminates my vLLM container after a certain (relatively short) time-period: ``` (VllmWorker rank=0 pid=783) Loading safetensors using Runai Model Streamer:...

Loading models from an S3 location instead of local path

@nijave wow that is quite hectic. This model is much larger 235B. I have my object store on NVMe in the same network, so its technically already on disk, I...

Loading models from an S3 location instead of local path

@nijave in my case this was actually just a healthcheck that failed - and it turned out to work perfectly without using sharding by simply setting the health and liveness...

Add ARM binaries to releases

argocd-image-updater commits all overridden argocd parameters

Same issue