fastertransformer_backend icon indicating copy to clipboard operation
fastertransformer_backend copied to clipboard

Published 20 hours ago • verified publisher icon triton-inference-server

enable llama model in FT backend

Open hongboshi1234 opened this issue 2 years ago • 1 comments

existing FT backend will throw error for llama model.

hongboshi1234 avatar Jun 24 '23 06:06 hongboshi1234

Will this ever work? I didn't see llama defined under: https://github.com/NVIDIA/FasterTransformer/tree/main/src/fastertransformer/triton_backend

sfc-gh-zhwang avatar Jul 08 '23 00:07 sfc-gh-zhwang