tiktoken under counting tokens for openai text-embedding-3-large

[mattkauffman23]

trafficstars

Starting on January 2nd 2025 we started noticing errors in our logs that we were over the context limit when creating text-embedding-3-large embeddings on openai. I believe there may have been a change on the openai side since we hadn't made any related changes. In one case I looked into tiktoken reported 7995 tokens but we received the following error from openai:

{'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 8781 tokens (8781 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}

For the moment we're reducing our limit for max tokens to mitigate, but wanted to raise the issue.