[PROBLEM] Tiktoken sample implementation missing latest models (since gpt-4-turbo)

[jalcantarab]

trafficstars

Identify the file to be fixed The name of the file containing the problem.

Describe the problem The current tiktoken sample implementation does not include the latest models released since gpt-4-turbo. These models, such as gpt-4o, gpt-4o-2024-05-13, and gpt-4-turbo-2024-04-09, have different token pricing and context limits that need to be accounted for.

Describe a solution Update the tiktoken sample implementation to include the latest models:

• gpt-4o
• gpt-4o-2024-05-13
• gpt-4-turbo-2024-04-09 Ensure their respective encoders, context limits, string checks are correctly implemented.