Bhimraj Yadav

Hi @elbamos, thanks for raising this! It might help to set up the dataset/dataloader through a Lightning DataModule to ensure everything gets initialized correctly across processes. Could you also share...

Hi @kumarrah2002 , About the exceptions during shutdown — I think it might be acceptable as long as we ensure that: No child processes or threads are left running and...

Hi @riyajatar37003, this studio might be helpful for your use case: https://lightning.ai/bhimrajyadav/studios/deploy-openai-like-embedding-api-with-litserve-on-studios. > It also includes additional features like support for different models. Feel free to modify it to suit...

Setting `persistent_workers=True` might help resolve the issue. If that doesn’t work, @kirk, @kirk, would you mind sharing the complete error traceback? That would really help us narrow down where exactly...

Hi @kyoungrok0517, thanks a lot for opening the issue! To help us look into it, could you also share a bit more detail about the dataset? If it’s publicly available,...

Thank you for sharing the details, @kyoungrok0517! I'll give it a try with the sample dataset and see if I can reproduce the issue on my end.

> [@bhimrazy](https://github.com/bhimrazy) I wonder if the issue isn t coming from the compression where we start reading from the file before it is fully there @tchaton Thanks for the suggestion!...

**Update on the DDP Test:** I've run tests under DDP using the provided sample data, as well as other datasets like `OpenThoughts-114k` and `fineweb-edu/sample/10BT`. So far, I haven’t been able...

> [@bhimrazy](https://github.com/bhimrazy) Thanks for trying :) I'll create and share a reproducible project on Lightning Studio soon. Thank you, @kyoungrok0517 — really appreciate it!

Hi @kyoungrok0517 👋 — just checking in to see if you’ve had a chance to create the Lightning Studio project that reproduces the issue. No rush at all—just wanted to...