GGBond8488

[Bug]: Lora cannot be loaded in API mode

19

### Is there an existing issue for this? - [X] I have searched the existing issues and checked the recent builds/commits ### What happened? when I run webui --api to...

ValueError: agread: bad input data, first read normal. second error, retry second normal

5

I have a batch processing task that will read batch dot files. Reading the first dot file is normal. When reading the second file, an error will be reported. However,...

【Don't Merge】Nsfnet run example

1

### PR types Others ### PR changes Others ### Describe ...

A template demonstrating how to test higher-order autodiff.

6

### PR Category Others ### PR Types Others ### Description pcard-81841 A template demonstrating how to test higher-order autodiff.

### System Info NVIDIA-H100 ### Who can help? @kaiyux ### Information - [X] The official example scripts - [ ] My own modified scripts ### Tasks - [X] An officially...

like the https://github.com/microsoft/mscclpp

Can tensorrt-llm or how tensorrt-llm support that seprating the prefill stage and decode stage in different GPU or different nodes with self configuration

2

such as https://github.com/vllm-project/vllm/pull/2809 and https://github.com/LLMServe/DistServe that had done reference:https://arxiv.org/pdf/2311.18677

deploy with the IFB, when user input the paylod as follow: ``` { "text_input": str(question), "max_tokens": 512, "bad_words": "", "stop_words": stop_words, "pad_id": pad_id, "end_id": end_id, "top_p": 1, "id": "ggbond_test", "temperature":...

Test llama ernie in dy2st prim cinn

3

### PR types ### PR changes ### Description

Does triton inference server support customers custom feature but do not need to modify the origin code, like some plugin feature?

2

**Is your feature request related to a problem? Please describe.** I have some special feature want to apply in triton and I don't want to modify the origin code directly,...