swift icon indicating copy to clipboard operation
swift copied to clipboard

Published 20 hours ago • verified publisher icon modelscope

关于训练数据集数量与steps的问题

Open piqiuni opened this issue 2 months ago • 2 comments

请问在微调中我们使用的数据集有29450条，batch_size=1, train_dataset_sample=-1, 为什么训练一个epoch使用约1800steps?

按我的理解batch_size=1每次加载一条数据，step应该与数据数量一致？

感谢您的回答！

piqiuni avatar Apr 26 '24 04:04 piqiuni

默认会设置梯度累加的, 使得total_batch_size >= 16

Jintao-Huang avatar Apr 26 '24 05:04 Jintao-Huang

Steps per epoch = Total number of samples / (batch size * gradient accumulation steps)

catundchat avatar Apr 28 '24 08:04 catundchat