jiahanc

icon company @Nvidia

Results 13 comments of jiahanc

fix：fix illeagel memory access when mtp >= 2

Hi @dongjiyingdjy I think the issue also exists when MTP =2, can we change the title to make it more clear? Thanks :)

How to reproduce 150 TPS using FP8 + MTP=0 + BSZ=1 on H200?

Hi @ghostplant , The 150 TPS is with MTP = 3. We have a PR to document the reproduction steps on both Hopper and Blackwell: https://github.com/NVIDIA/TensorRT-LLM/pull/3232

[API change] Allow using torch.Tensor for scales for trtllm-gen attention

/bot run

[API change] Allow using torch.Tensor for scales for trtllm-gen attention

/bot run

[API change] Allow using torch.Tensor for scales for trtllm-gen attention

/bot run

[API change] Allow using torch.Tensor for scales for trtllm-gen attention

/bot run

[API change] Allow using torch.Tensor for scales for trtllm-gen attention

/bot run

fix: Reverse graph size order

/bot run

fix: Reverse graph size order

/bot run

fix: Reverse graph size order

/bot run