Richard Ginsberg

Ollama v0.1.18+ does not fully unload from GPU when idle

10

**OS:** Ubuntu 22.04 **Environment:** Docker/nvidia container **Server:** Dell Poweredge R720 **GPUs:** Nvidia Tesla P40 24GB **GPU quantity:** 2 **Model:** any (ie. dolphin-mixtral:8x7b-v2.5-q6_K) ``` docker pull ollama/ollama:0.1.17 docker run -d --gpus=all...

fastchat.serve.openai_api_server doesn't work with `stream=true` parameter

2

I'm trying to get FastChat working on https://github.com/open-webui/open-webui via it's option to accept OpenAI compatible endpoints. TabbyAPI seems to be able to serve OpenAI API format with streaming/chunking. FastChat doesn't...

Typos