FasterTransformer icon indicating copy to clipboard operation
FasterTransformer copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

TP=2， Loss of accuracy

Open coderchem opened this issue 2 years ago • 2 comments

hello,I carried out TP=2,multi-gpu operation on llama model of 7b, and found that the comparison accuracy of the result was lost by 5%. TP=2 as far as I know should not change the accuracy. Why?

coderchem avatar Jul 28 '23 03:07 coderchem

hi, can you post reproduce step.

hurun avatar Jul 29 '23 10:07 hurun

FasterTransformer does not support llama officially and FasterTransformer development has transitioned to TensorRT-LLM. TensorRT-LLM has supported LLaMa, please take a try.

byshiue avatar Oct 20 '23 07:10 byshiue