litdata
litdata copied to clipboard
Lightning-AI
Question: is there a plan to support streaming from GCS?
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Hey @dnnspark,
Yes, this is quite simple to add. Simply needs to add the downloader.
Thanks @tchaton, do you have an idea when it's going to land (even very rough estimate)?
Hey @dnnspark,
If you are willing to give it a try, I can look into it this week.
Sorry for the late @tchaton
I'm willing to try! But it's not blocking at the moment, so I will stay tuned about the GCS support (it will be very helpful if you ping on this thread once it's ready).
One thing I notice is that optimize() function assumes the data is stored on local disk (at least in the example). In my case, the raw data is at GCS (because it's too large). Is there a way to transform the data that is stored in the cloud, and save the transformed data to the cloud without having to download the entire data?
Sorry for the late @tchaton
I'm willing to try! But it's not blocking at the moment, so I will stay tuned about the GCS support (it will be very helpful if you ping on this thread once it's ready).
One thing I notice is that
optimize()function assumes the data is stored on local disk (at least in the example). In my case, the raw data is at GCS (because it's too large). Is there a way to transform the data that is stored in the cloud, and save the transformed data to the cloud without having to download the entire data?
Yes. that's why this library was built :) But I would need to add GCS support for it ;) I will try to prioritize it.