Token Count Calculation in SFT Data Distribution Curation

[tcy6]

Regarding the curation of SFT data, including the curation of the data distribution, I would like to understand how you calculate the token count for each data entry when designing the distribution. Is the token count based only on the user tokens, or does it also include the assistant tokens? (The reason I ask is that I understand the SFT loss is calculated only based on the assistant tokens.)

[Balakrishna-Chennamsetti]

Hi @tcy6 ,

The token count isn't one number. Both counts are calculated and used. When considering training cost and efficiency, token count usually refers to the total tokens. When considering task balancing and model behavior, it refers to the assistant tokens.

1. Assistant-Only Tokens: This metric measures the length of the assistant's response (len(assistant_response)). Its primary purpose is for Learning & Task Balancing answering the question.
2. Total Tokens: This metric measures the combined length of the user prompt and the assistant's response (len(user_prompt) + len(assistant_response)). Its primary purpose is for managing Infrastructure & Cost answering the question.

Thanks.