openai-cookbook
openai-cookbook copied to clipboard
Update tiktoken models with gpt-4-turbo and gpt-4o
trafficstars
Summary
This PR updates the How_to_count_tokens_with_tiktoken.ipynb notebook to include the latest models released since gpt-4-turbo. The updates ensure compatibility with gpt-4-turbo, gpt-4o, and their respective variants, along with small corrections on the number of encoders and examples.
Motivation
These changes are necessary to keep the notebook up-to-date with the latest OpenAI models. By including the new models and their respective encodings, users can accurately count tokens for all supported models, enhancing the utility and accuracy of the notebook.
For new content
When contributing new content, read through our contribution guidelines, and mark the following action items as completed:
- [x] I have added a new entry in registry.yaml (and, optionally, in authors.yaml) so that my content renders on the cookbook website.
- [x] I have conducted a self-review of my content based on the contribution guidelines:
- [x] Relevance: This content is related to building with OpenAI technologies and is useful to others.
- [x] Uniqueness: I have searched for related examples in the OpenAI Cookbook, and verified that my content offers new insights or unique information compared to existing documentation.
- [x] Spelling and Grammar: I have checked for spelling or grammatical mistakes.
- [x] Clarity: I have done a final read-through and verified that my submission is well-organized and easy to understand.
- [x] Correctness: The information I include is correct and all of my code executes successfully.
- [x] Completeness: I have explained everything fully, including all necessary references and citations.