Sam Hjelmfelt

icon indicating a website link https://www.linkedin.com/in/samhjelmfelt

icon location Remote

Results 1 comments of Sam Hjelmfelt

Waiting sequence group should have only one prompt sequence.

I encountered this with Mistral 7b on an A10 using AsyncLLMEngine when pending requests increased above 0. Removing n and best_of from the SamplingParams is a workaround.