RobinJYM

ChatGLM3 6B Multi-batch Failed with Error

2

### System Info - CPU: INTEL RPL - GPU Name: NVIDIA GTX 4090 - TensorRT-LLM: tensorrt_llm==0.11.0.dev2024060400 - Container Used: Yes and reproduced in Conda as well - Driver Version: 555.42.02...

How to identify the rest toke latency?

3

### System Info - CPU: INTEL RPL - GPU Name: NVIDIA GTX 3090 - TensorRT-LLM: tensorrt_llm==0.11.0.dev2024060400 - Container Used: Yes and reproduced in Conda as well - Driver Version: 555.42.02...