FasterTransformer icon indicating copy to clipboard operation
FasterTransformer copied to clipboard

[Bugfix] GptJ & GptNeoX batch inference error

Open YZP17121579 opened this issue 2 years ago • 1 comments
trafficstars

GptJ & GptNeoX may generate random outputs when using batch inference mode and no prefix prompt. The problem is caused by the nullptr check in https://github.com/NVIDIA/FasterTransformer/blob/f8e42aac45815c5be92c0915b12b9a6652386e8c/src/fastertransformer/kernels/gpt_kernels.cu#L1064

YZP17121579 avatar Aug 11 '23 09:08 YZP17121579

I think this is a duplicate solution of #716 which is more elegant and efficient.

BasicCoder avatar Aug 12 '23 01:08 BasicCoder