Bhimraj Yadav comments

Results 168 comments of


                                            Bhimraj Yadav

Investigate keeping the content of the downloaded chunks in RAM instead of writing it to file.

**Update** I prototyped an in-memory version of `PyTreeLoader`, similar to our earlier streaming ideas. It downloads chunks into RAM and streams them as they arrive—no file writes. Tested on 12GB...

Using a streaming dataloader with an unbalanced dataset yields unexpected batch sizes.

Lets keep this. Worth checking at somepoint.

Add support for the reduce operator

Let's keep this — it's worth exploring and discussing it.

Profiler patches torch without reverting it

@Borda Currently, we have a check in place to patch only when profiling is enabled: https://github.com/Lightning-AI/litData/blob/46a40496a354555e6f3c8a18f157583a92830efb/src/litdata/streaming/dataloader.py#L473-L478 That said, it might still be worth revisiting this to find a cleaner or...

LitData doesn't support s3 bucket connection outside server

With `storage_options` support in LitData, unsigned S3 requests can be handled easily by passing a custom `config` via `storage_options`. Example: ```python import botocore from litdata import StreamingDataset storage_options = {...

If data samples contain a Python list with a variable number of elements, type inference will fail

Let's keep this , worth exploring.

Data shard delation with multi GPU does not work

Is this issue still relevant?

Data shard delation with multi GPU does not work

Thanks @rakro101 — appreciate the update! Looking forward to it.

Evict requests if the client has disconnected

Hi @aniketmaurya, I've been exploring potential implementations for this feature and came across the [req.is_disconnected()](https://fastapi.tiangolo.com/reference/request/#fastapi.Request.is_disconnected) method, which seems useful for checking if the client has disconnected. I will try to...

Evict requests if the client has disconnected

Thank you, @aniketmaurya, for the insights, and apologies for the delay in drafting the PR. I've tried implementing an idea by checking for the uid. However, I'm a bit unsure...