jan icon indicating copy to clipboard operation
jan copied to clipboard

feat: enable `continue` button for when LLM responses exceed token limit

Open dan-jan opened this issue 8 months ago • 11 comments

Describe the bug LLM responses seem to terminate prematurely, do we have a max reply token setting somewhere? EDIT: Mistral likely limited to 512?

To Reproduce Steps to reproduce the behavior:

  1. Use Mistral 7b instruct
  2. Ask Mistral 7b to generate 100 names similar to "Thinking Machines"
  3. Mistral will generate 63 or so
  4. Ask Mistral to continue
  5. Mistral generates the remaining 100

Expected behavior Mistral should generate the full 100

Screenshots image

image

Desktop (please complete the following information):

  • OS: Mac M2 64GB RAM
  • Browser [e.g. chrome, safari]
  • Version v0.2.0

Additional context Add any other context about the problem here.

dan-jan avatar Oct 19 '23 14:10 dan-jan