jan
jan copied to clipboard
feat: enable `continue` button for when LLM responses exceed token limit
Describe the bug LLM responses seem to terminate prematurely, do we have a max reply token setting somewhere? EDIT: Mistral likely limited to 512?
To Reproduce Steps to reproduce the behavior:
- Use Mistral 7b instruct
- Ask Mistral 7b to generate 100 names similar to "Thinking Machines"
- Mistral will generate 63 or so
- Ask Mistral to continue
- Mistral generates the remaining 100
Expected behavior Mistral should generate the full 100
Screenshots
Desktop (please complete the following information):
- OS: Mac M2 64GB RAM
- Browser [e.g. chrome, safari]
- Version v0.2.0
Additional context Add any other context about the problem here.