microsoft
Token Limit Error in logs.json (Constant Completion Tokens)
-
Whatever changes i tried to do in settings.yaml file, I always get token limit error. Input token ranging from 2000-2300, but Completion token is fixed to 2000 always. How can i reduce the completion token size or input token size as well if you have any idea?? the text file is not that large as well. Contains only a few paragraphs of texts.
-
Tried changing the max token values and played around with the tokens_per_minute and requests_per_minute and still didnt get anywhere.
-
This is the current params I've used. Tried with different values as well but ended with same error. max_tokens: 290 tokens_per_minute: 1 # set a leaky bucket throttle requests_per_minute: 1 # set a leaky bucket throttle max_retries: 5 max_retry_wait: 120.0 concurrent_requests: 10 # the number of parallel inflight requests that may be made
Please help me out if you have faced any issue like this. or how to solve this if you have any ideas. Thank you!
Hi @Karthik-277 This was caused due to a bug on configuration that was ignoring settings for tpm and rpm, this has been fixed in this PR: #373
If you're using the source, we recommend pulling the latest main, if not, the fix will be included in our next release. Will keep this issue open until a new release is done
