openai-cookbook
openai-cookbook copied to clipboard
Update gpt-3.5-turbo token limit in Chat_finetuning_data_prep.ipynb
trafficstars
Summary
The token limit in Chat_finetuning_data_prep.ipynb is not up to date with the current context window limit. It should be 16,385 tokens as stated in https://platform.openai.com/docs/models/gpt-3-5-turbo
Motivation
This notebook gives a false warning since examples greater than 4096 tokens but less than 16,385 tokens will not be truncated.