thomas chaton

Hey @ouj, interesting. You have a slightly different issue. This issue was tracking increasing time during optimize, not in the StreamingDataset. So it would be better for you to open...

> I might find an opportunity to @tchaton . Is there a particular example that the team has used with synthetic data that I can use as a counter example...

Hey @sritterginkgo. Thanks for the update. If you have time, we can schedule a call to pair debug it and hopefully, find the source.

Hey @sritterginkgo. That's awesome. Do you want to make a PR to fix it ? Some context: The main reason why there were this loop in the first place was...

Hey @sritterginkgo, go ahead with submitting a PR and adding me for review. Make sure to add some tests too ;) > Do you or the team have some document...

Hey @sritterginkgo Actually, I think my code is wrong anyway because the min_index doesn't start from 0 and it is based on the received indexes. So you receive, 2, 4,...

Closing from https://github.com/Lightning-AI/litdata/pull/146

Hey @enrico-stauss, can you share a reproducible script of the problem. Not sure I fully follow it. The size of the StreamingDataset should be exact. If not, there is a...

Hey @enrico-stauss, changing the base type is a very large task and not something I am planing to do.

@enrico-stauss I think I have a fix. Could you try this branch: https://github.com/Lightning-AI/litdata/pull/139. This will work only with the StreamingDataLoader Example of the issue: There is 300 samples, 2 workers,...