Using fsspec to download files

[samsja]

🚀 Feature

I was wondering if you planned on using fsspec for downloading file. Since fsspec is already compatible with many cloud provider it would make lit-data streaming compatible with even more cloud.

Motivation

Right now there is one downloader class for each provider, fsspec would allow to abstract it away, reducing the amount of cloud provider specific code and allowing supporting more out of the box

PS: love the work on lit data

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[tchaton]

Hey @samsja, feel free to contribute fsspec support

[samsja]

Hey @samsja, feel free to contribute fsspec support

Sure, I can give it a try

[deependujha]

Hey @samsja, there has been no update from your side regarding any progress.

So, I'd like to work on this.

If you have any issue, plz let me know.

[samsja]

Hey @samsja, there has been no update from your side regarding any progress.

So, I'd like to work on this.

If you have any issue, plz let me know.

Unfortunately, I don't have time right now to check this. Feel free to take it

[deependujha]

For better fine-grained control and to support byte-range downloading and other powerful features, we've decided not to use fsspec and rather use our own implementation for each provider.

The first step towards this is with adding support for gcp: #504

Thanks. Closing the issue. Feel free to open if you've any other thoughts.