framework icon indicating copy to clipboard operation
framework copied to clipboard

Published 20 hours ago • verified publisher icon frictionlessdata

Review if we can use `fsspec` for data loading

Open roll opened this issue 2 years ago • 5 comments

Overview

https://pypi.org/project/fsspec/

roll avatar Mar 13 '23 09:03 roll

Heya @roll! I'm curious if you can expand on this one. Are you thinking on using fsspec to replace the scheme parts of the codebase?

I've been thinking on the best way to read/write data on a non existing scheme and thinking about this these days.

davidgasquez avatar Mar 22 '23 18:03 davidgasquez

@davidgasquez Yes, we historically used these loaders approach created for tabulator, but I'm curious if fsspec can handle it (I haven't dove into it yet unfortunately)

roll avatar Mar 23 '23 10:03 roll

I think it should!

A potentiall intermediary approach might be to add a fsspec scheme via plugins. Does this makes sense?

davidgasquez avatar Mar 23 '23 10:03 davidgasquez

TBH, currently, I'm not sure as it seems fsspec might just a perfect replacement for the whole Loaders part making the parsers and the framework way simpler.

We're going to work soon on the v6 - https://github.com/orgs/frictionlessdata/projects/16/views/11 - which will include optimizations for the whole reading flow. Might be a good chance for the fsspec migration

roll avatar Mar 27 '23 13:03 roll