thomas chaton comments

Results 291 comments of


                                            thomas chaton

The config isn't consistent between chunks

Yes, LitData encodes each leaf of the pytree as a single object and therefore, it doesn't know this is a single sample. You can convert it to numpy or torch...

The config isn't consistent between chunks

Hey @gluonfield Litdata supports parquet files now. So u don t need to optimise it anymore

Add support for configuring the upper directory of a docker container under a given directory

Any updates ? This is quite important for us. I have a solution to make it work, but it is racy. I can also work on contributing this.

Add support for configuring the upper directory of a docker container under a given directory

Hey @thaJeztah @GordonTheTurtle Any updates ?

Add support for configuring the upper directory of a docker container under a given directory

Thanks @vvoland . Do you have docs to create a custom snapshotter ?

Add support for configuring the upper directory of a docker container under a given directory

Hey @vvoland Maybe this would work: https://github.com/containerd/containerd/blob/c673fcc9af3ec4a44ac112bd1e8e6479862a10af/docs/snapshotters/README.md?plain=1#L42C8-L45C27

Feat: configure logging. Add support for easy logging setup with sens…

Hey @sanggusti. Would you mind adding a bit more info to the description. Also, it seems the tests are failing.

A contributing.md for the project

We can adapt: https://github.com/Lightning-AI/pytorch-lightning/blob/master/.github/CONTRIBUTING.md

CONTRIBUTING.md for LitData

Nice @deependujha !

How can I shut down automatically distributing data when using StreamingDataset?

Hey @ygtxr1997. You can override the distributed env on the dataset. It is inferred automatically from torch. What is your use case ?