llm-inference icon indicating copy to clipboard operation
llm-inference copied to clipboard

Expose model generate parameters by API server

Open HaiHui886 opened this issue 1 year ago • 0 comments

generate_kwargs:
  do_sample: true
  max_new_tokens: 128
  min_new_tokens: 16
  temperature: 0.7
  repetition_penalty: 1.1
  top_p: 0.8
  top_k: 50

HaiHui886 avatar Mar 27 '24 13:03 HaiHui886