torchtitan icon indicating copy to clipboard operation
torchtitan copied to clipboard

[Feature] More pythonic approach to dataloading

Open CarlosGomes98 opened this issue 8 months ago • 0 comments

Hey Folks!

I've had a really good time playing with torchtitan so far :)

Looking into the code, as it stands, the way the data loader is wrapped by next_batch is not very pythonic. It makes it very awkward to iterate through the data loader in a pythonic way with for or while.

#1138 is my draft-suggestion of a way to pythonify this.

I think this will especially bear fruit in situations where we might want to iterate through a dataset from beginning to end, e.g., for a validation dataset.

To note, this current change might break a few of the models under experimental, which would need to mirror this change in their train.py

Just wanted to start this discussion. I know its in a particularly critical part of the architecture, so I understand friction regarding changes to it

CarlosGomes98 avatar Apr 23 '25 15:04 CarlosGomes98