text-generation-inference

text-generation-inference copied to clipboard

Improve inference speed of Santacoder and Starcoder (and others)

7

I did some extensive investigation, testing and benchmarking, and determined that the following is needed to speedup inference for the Bigcode models (and most of text-gen-inference models: 1. **Use `FlashAttention...

jlamypoirier

Adapter support

1

### Feature request Enable the use of locally stored adapters as created by huggingface/peft. Ideally, this should be compatible with the most notable benefits of TGI (e.g. sharing and flash...

QLutz

feat(router): Dynamic batch sizing

4

#### Motivation Currently to avoid OOM you must set a "worst case" max batch size based on the desired max sequence length. This means that (a) throughput is unnecessarily limited...

njhill

How do i use `trust_remove_code=True` for mosaic models?

6

### Feature request ``` pretrained_model_dir = 'mosaicml/mpt-7b' model = AutoModelForCausalLM.from_config(config, trust_remote_code=True, torch_dtype=torch.float16) ``` https://discuss.huggingface.co/t/how-to-use-trust-remote-code-true-with-load-checkpoint-and-dispatch/39849/1 ### Motivation model-specific params ### Your contribution sure

cfregly

Prototype to greatly reduce the post-processing overhead at higher batch sizes.

jlamypoirier

# What does this PR do? Reworked the loading logic. Idea is to use cleaner loading code: - Remove need for `no_init_weights` - Remove all weird `bnb_linear` and `load_weights` and...

Narsil

[WIP] Adding GPTQ support for llama

3

# What does this PR do? Fixes # (issue) ## Before submitting - [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks...

Narsil

[Meta] Road to 1.0 checklist

11

### CI - [x] Add custom multi GPU runner to CI - [x] test docker image on MR - [x] load test on daily cron (low prio) ### server -...

Narsil

How to import chatglm model

2

### System Info text-generation-inference: v0.7.0 python: 3.9 Operation System: Ubuntu 18.04 When loading chatglm model use command: docker run --gpus '"device=3"' --shm-size 1g -p 8083:80 -v /data/llm:/data ghcr.io/huggingface/text-generation-inference:latest --model-id /data/chatglm-6b...

askintution

Downloading stuck for some models

4

### System Info Using latest docker ### Information - [X] Docker - [ ] The CLI directly ### Tasks - [X] An officially supported command - [ ] My own...

d0lphin