openai-cookbook icon indicating copy to clipboard operation
openai-cookbook copied to clipboard

Update gpt-3.5-turbo token limit in Chat_finetuning_data_prep.ipynb

Open seandiacono opened this issue 1 year ago • 0 comments
trafficstars

Summary

The token limit in Chat_finetuning_data_prep.ipynb is not up to date with the current context window limit. It should be 16,385 tokens as stated in https://platform.openai.com/docs/models/gpt-3-5-turbo

Motivation

This notebook gives a false warning since examples greater than 4096 tokens but less than 16,385 tokens will not be truncated.

seandiacono avatar May 08 '24 11:05 seandiacono