tiktoken
tiktoken copied to clipboard
fix: add encoding for fine-tuned models based on gpt-4o
trafficstars
The encoding is different for gpt-4o than gpt-4 models. The current implementation matches "ft:gpt-4": "cl100k_base" in the MODEL_PREFIX_TO_ENCODING dictionary for fined-tuned models based on gpt-4o.