llama-stack

llama-stack copied to clipboard

# Changes ### evals API - skeleton evals API using inference, synchronous returning `EvaluateResponse` ### task/dataset registry - `TaskRegistry` for looking up eval tasks - `DatasetRegistry` for looking up datasets...

yanxi0830

Add Support for Kubernetes

1

As of now, Llama Stack only supports single node providers for the self-hosted solutions. Is it possible to support Kubernetes as a provider or some other architecture to make Llama...

jdelarosaquiros

Cannot run llama-stack on windows due to termios dependency -- Build fails with poetry

2

I used poetry to add llama-stack, executing the basic build from the quick guide gives me "No module named 'termios'. Running on a windows machine.

ShadiCopty

ollama inference should verify models are downloaded before serving

1

Right now, if you start a distribution using `remote::ollama` without the models downloaded, ollama will attempt to download it upon the first inference request: ``` INFO: Uvicorn running on http://[::]:5001...

dltn

Configuring ollama build with redis results in an error. llama stack run ollama-stack --port 5000 Resolved 8 providers in topological order Api.models: routing_table Api.inference: router Api.shields: routing_table Api.safety: router Api.memory_banks:...

clearstorm-tech

Are there any available tools that can convert the original .pth to safetensors

2

Are there any available tools that can convert the original .pth model files downloaded from Meta into a format usable by stack, or convert them to .safetensors format? I tried...

Itime-ren

stack tool cannot support large models with a .pth extension downloaded from Meta

2

The stack tool cannot support large models with a .pth extension downloaded from Meta. It throws an error during runtime. Does it have to use models downloaded from Hugging Face?...

Itime-ren

Create shared openai-compatible inference adapter

2

There are multiple open PRs looking to add inference adapters for services that offer OpenAI-compatible APIs. * databricks - https://github.com/meta-llama/llama-stack/pull/83 * sambanova - https://github.com/meta-llama/llama-stack/pull/136 * runpod - https://github.com/meta-llama/llama-stack/pull/157 * vllm...

russellb

Add generic OpenAI compatible inference provider

5

This PR is refactored from https://github.com/meta-llama/llama-stack/pull/178 to a more generic OpenAI compatible inference provider. Other remote providers that are compatible to OpenAI protocols (e.g. https://github.com/meta-llama/llama-stack/pull/83, https://github.com/meta-llama/llama-stack/pull/136, https://github.com/meta-llama/llama-stack/pull/157, https://github.com/meta-llama/llama-stack/pull/178) should be...

terrytangyuan

Question regarding agents and sessions APIs

2

Hello, I have a question regarding the API with Agents. I did not find a way to get an agent that is already created. Or list the agents available. Did...

Minutis