griptape icon indicating copy to clipboard operation
griptape copied to clipboard
verified publisher icon griptape-ai

Remove Tokenizer Output Token Tracking

Open collindutter opened this issue 9 months ago • 0 comments

Griptape's Tokenizers maintain two maps: MODEL_PREFIXES_TO_MAX_INPUT_TOKENS and MODEL_PREFIXES_TO_MAX_OUTPUT_TOKENS. The former is used when performing operations that try to maximize the LLM's context window, like summarization. The latter, is an artifact of when we used to calculate output tokens with OpenAi to avoid rate limiting. This was removed quite a while ago and the functionality is not used anywhere else in the framework.

Side effect of it existing is:

  • Maintenance on this token mapping.
  • Messiness with how to treat embedding models (#1842).

The functionality should be removed.

collindutter avatar Mar 17 '25 16:03 collindutter