FasterTransformer icon indicating copy to clipboard operation
FasterTransformer copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

decoupled mode not working when beam_width > 1

Open flexwang opened this issue 1 year ago • 0 comments

I am running t5 decoder model with fastertransformer, it seems that, if I set beam_width>1, the result that is stream back are just garbage tokens. Is this expected?

flexwang avatar Aug 10 '23 04:08 flexwang