tiktoken icon indicating copy to clipboard operation
tiktoken copied to clipboard

fix: add encoding for fine-tuned models based on gpt-4o

Open hughcrt opened this issue 1 year ago • 0 comments
trafficstars

The encoding is different for gpt-4o than gpt-4 models. The current implementation matches "ft:gpt-4": "cl100k_base" in the MODEL_PREFIX_TO_ENCODING dictionary for fined-tuned models based on gpt-4o.

hughcrt avatar Aug 01 '24 21:08 hughcrt