text-generation-inference

text-generation-inference copied to clipboard

Shard 2 failed to start

6

### System Info i was trying to run CohereForAI/c4ai-command-r-v01 with these commands model= CohereForAI/c4ai-command-r-v01 volume=$PWD/data # share a volume with the Docker container to avoid downloading weights every run docker...

yasserkh2

### System Info Docker image: TGI 1.4.5 Target: x86_64-unknown-linux-gnu Cargo version: 1.75.0 Commit sha: 4ee0a0c4010b6e000f176977648aa1749339e8cb Docker label: sha-4ee0a0c nvidia-smi: N/A ### Information - [X] Docker - [ ] The CLI...

seongminp

abhishekkrthakur

about python server

1

i want to use in python but it so slow, i want to use quickly

Soulscb

Fp6 quant

2

### Feature request The fp6 quant announced by deepspeed ### Motivation Lower latency, higher throughout, less memory ### Your contribution https://github.com/usyd-fsalab/fp6_llm/blob/main/tests/python/kernel_test.py Seems to be a low hanging fruit Can open...

flozi00

ExllamaV2 continuous batching supported?

1

### Feature request This is just a question (if no, then it is a request): Does Exllama V2 support have continuous batching? That's the only thing I find missing in...

SinanAkkoyun

Top-p sampling returns incorrect/missing log-probabilities in details

3

### System Info I noticed text-generation-inference is providing incorrect log-probabilities in the details whenever top-p sampling is used. I'm running text-generation-inference via docker, using this k8s pod definition on a...

kingb12

TGI support fo LLava models

1

### System Info version latest, sha-7dbaf9e ### Information - [X] Docker - [ ] The CLI directly ### Tasks - [X] An officially supported command - [ ] My own...

vitalyshalumov

Template Not Found When Using OpenAI format Chat Completion

23

### System Info Docker Image: ghcr.io/huggingface/text-generation-inference:sha-1734540 Instance: AWS A10G via Huggingface Interfence Endpoint ### Information - [X] Docker - [ ] The CLI directly ### Tasks - [X] An officially...

binarycrayon

AsyncInferenceClient - Unclosed client session

3

I'm using TGI with Flan-T5 to process thousands of text extraction requests at a time, on a 4 x A6000 machine. My client class, which uses `AsyncInferenceClient`, can handle 900...

mhillebrand