Is beam search strategy mandatory in gptj-6b benchmark?

[phoenixysj]

hi, I found that vllm will drop beam search feature in future releases due to complicated implementation and beam search is not so popular in real use. So could I know if gptj-6b benchmark test will drop beam search strategy too?

[psyhtest]

Yes, it is mandatory for the Closed division.

However, for the reasons that you outlined, GPTJ might be dropped from MLPerf Inference too. (Normally, a benchmark need to survive 4 submission rounds. Conveniently, GPTJ will make its 4th appearance in the upcoming v5.0 round.)