batch inference

[jtan1102]

Hi authors,

I want to test the performance of the Mistral7B on the test dataset. Is it only possible to do single sample inference (with model. generate(...))? Are there any methods to accelerate the process?

Thanks

[THUchenzhou]

You can use: for input_ids, output_ids in zip(batched_inputs.input_ids, batched_outputs):

or refer to https://github.com/ggerganov/llama.cpp