localGPT icon indicating copy to clipboard operation
localGPT copied to clipboard

Published 20 hours ago • verified publisher icon PromtEngineer

Requested tokens exceed context window of 4096

Open fenry46 opened this issue 1 year ago • 3 comments

So when I run the localGPT is fine. but when I try to run_localGPT_API when I try to ask question it's always throw exceed context window error is there anyone how to solve this issue ?

fenry46 avatar Jan 11 '24 08:01 fenry46

Change it to a model that supports 8k or 16k tokens such as zephyr or Yi series. Also you will need to change the max tokens here

PromtEngineer avatar Jan 12 '24 06:01 PromtEngineer

Thanks, I already try to change the model , but sometimes the answer is blank. Do you have any recommend local model that support 8k or 16k because when I find the zephyr and Yi series the GGUF one, the answer is blank.

fenry46 avatar Jan 22 '24 02:01 fenry46

@PromtEngineer Wouldn't changing the models will result into more hardware resources being consumed? Right? @fenry46 @PromtEngineer

Tell me more about it, please.

NitkarshChourasia avatar Mar 31 '24 22:03 NitkarshChourasia