text-generation-inference

text-generation-inference copied to clipboard

fp8 weight load failed IndexError: list index out of range

2

### System Info ``` 2024-08-13T06:17:44.049654Z ERROR shard-manager: text_generation_launcher: Shard complete standard error output: 2024-08-13 06:17:41.545 | INFO | text_generation_server.utils.import_utils::75 - Detected system cuda /opt/conda/lib/python3.10/site-packages/text_generation_server/utils/sgmv.py:18: UserWarning: Could not import SGMV kernel...

icyxp

RuntimeError: "weight lm_head.weight does not exist" When Loading qwen2-0.5B-Instruct

3

I'm experiencing an issue when loading the qwen2-0.5B-Instruct model with the TGI library. The error message thrown is "RuntimeError: weight lm_head.weight does not exist". I suspect this might be due...

boyang-nlp

### System Info Docker image: `ghcr.io/huggingface/text-generation-inference:2.2.0` GPU: T4 OS: Ubuntu 20.04.6 LTS ### Information - [X] Docker - [ ] The CLI directly ### Tasks - [X] An officially supported...

ilyalasy

InternLM2.5 support

1

### Model description TGI 2.2.0 doesn't support the InternLM2.5 model family, which is supported by transformers. ### Open source status - [X] The model implementation is available - [X] The...

juliensimon

Disable logging of "grammar" parameter

1

### Feature request Remove full logging of the "grammar" parameter on each request. Possible replacements: a) log only "None" vs "Some" b) add a config argument to enable/disable full logging...

sfbemerk

Build Intel CPU optimized image automatically

1

Hello, we are looking for the best way for deploying TGI on Xeons. I understand that container images tagged with `x.y.z-intel` are the XPU builds, while `Dockerfile_intel` defines both XPU...

Feelas

Recent issues building text-generation-server with torch+cu118

3

### System Info tgi version >=2.2.0 python 3.9 cuda 11.8 torch 2.4.0+cu118 (also 2.3.0+cu118) ### Information - [ ] Docker - [X] The CLI directly ### Tasks - [X] An...

tderrmann

feat: add release and sha tagged images

1

This PR adds a new image tag to include both the latest version and sha tag on images. This is helpful in understanding which version the latest changes derive from

drbh

Newer HF Mamba model is not supported

1

### System Info TGI v2.2.0 with the official Docker image. ### Information - [x] Docker - [ ] The CLI directly ### Tasks - [x] An officially supported command -...

jonnyli1125

TGI does not support DeepSeekCoderV2-gptq

2

### System Info A100-80GB * 4 ### Information - [X] Docker - [ ] The CLI directly ### Tasks - [X] An officially supported command - [ ] My own...

Grey4sh