text-generation-inference issues

feat(server): have FlashGPTNeoXModel support HF accelerate

It seems to work fine and loads 4-10x faster for me depending on the storage/page cache (non-sharded 20B parameter model). However, when loaded this way, inference appears to be 10-15%...

njhill

proposal: Move token decoding and stopping evaluation to router

4

Benefits: - Centralizes this logic that's on the critical inference loop path and does it in rust instead of python - Simplifies python side of the code, decoupling next-token generation...

njhill

Expose GPU, CPU and memory utilization as prometheus metrics

### Feature request In addition to per request metrics such `tgi_request_count` it would be useful to have utilization metrics returned on a per interval basis similar to the way Triton...

brightsparc

Explain the generation parameters in the benchmarking utility

8

### Feature request Improved `README.md` for the benchmarking utility that explains the different command line arguments. ### Motivation The benchmarking tool is awesome, I would just like to have some...

Blair-Johnson

Error deploying docker on P3 instance

2

### System Info ```bash docker run --gpus all --shm-size 1g -p 8080:80 -v $PWD/data:/data ghcr.io/huggingface/text-generation-inference:0.8 --model-id tiiuae/falcon-7b --num-shard 2 ``` ``` ubuntu@ip-172-31-35-173:~$ nvidia-smi Wed Jun 7 09:30:53 2023 +-----------------------------------------------------------------------------+ |...

FionnD

Segmentation fault when downloading models

3

### System Info ``` 2023-06-07T08:37:39.808440Z INFO text_generation_launcher: Runtime environment: Target: x86_64-unknown-linux-gnu Cargo version: 1.69.0 Commit sha: 19c41824cb11ba1a3b60a2a65274d8c074383de3 Docker label: N/A nvidia-smi: Wed Jun 7 17:37:39 2023 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 515.65.07...

jshin49