remmen-io comments

Repositories
Issues
Comments

Results 3 comments of


                                            remmen-io

Failed to download model with docker behind proxy

Simply download your model to a location (here `models-cache`) and set `HUGGINGFACE_HUB_CACHE=/models-cache` and `MODEL_ID=/models-cache/deepseek-coder-33B-instruct-AWQ`

[Feature]: Add a custom AI that can call rest API endpoint

I've tried to setup the localai to point to a local endpoint made with huggingface tgi. ``` k8sgpt auth update localai --model tgi --baseurl https://deepseek.k8scluster.ch/v1 ``` but I get a...

Truncated output with local backend?

Hi @ido50 Unfortunately no as TGI currently is not supporting this on the server side There is a open issue: https://github.com/huggingface/text-generation-inference/issues/870