llm.c icon indicating copy to clipboard operation
llm.c copied to clipboard

Possible bugs in the data loading functions

Open PeterZhizhin opened this issue 10 months ago • 1 comments

First, we read B*T+1 tokens, but advance the iterator by B*T tokens instead.

Then, there is this if statement:

    if (loader->current_position + (loader->num_processes * B * T + 1) * sizeof(int) > loader->file_size)

Possibly, we should remove the loader->num_processes multiplication here.

We need to verify that this is the way these functions should work.

PeterZhizhin avatar May 01 '24 10:05 PeterZhizhin

Hey @PeterZhizhin feel free to close this issue, the +1 is not a bug because it is used only in target when you load a first batch and in the next batch it'll be part of the input and not target, so it's actually fine.

gordicaleksa avatar Jun 07 '24 14:06 gordicaleksa