General question about project interaction with fsspec?

[yohplala]

Hi, I am discovering this project and would like to know about its status? It is presented as a fsspec implementation. Is it connected to fsspec and can files be managed on Google Drive by using fsspec? With fsspec, can we pip install fsspec[gdrive] ?

I have been working with Google Drive API v3 a while ago, relying on service account to provide a connector to Google Drive within cryptostore project (you can see the documentation page here). Is gdrivefs able to manage connection by service account? (wondering if this connector I did could not be replaced with fsspec / gdrivepec?)

Thanks in advance for any feedbacks! Bests

[martindurant]

This repo was indeed designed to be the gdrive implementation for fsspec. It is incomplete and untested, but seems to work for at least some uses. Indeed, we don't really know how to set up any kind of tests, except for vcrpy (as gcsfs uses), which is a painful process. gdrive is not really mass storage

With fsspec, can we pip install fsspec[gdrive]

No, this is not in the options list for fsspec, because of its very alpha status (although it is listed in the fsspec docs).

Is gdrivefs able to manage connection by service account?

I don't know! Certainly, if you have code that can authenticate using service accounts, it would be a useful addition here.

[yohplala]

Thanks for your feedback @martindurant !

Certainly, if you have code that can authenticate using service accounts, it would be a useful addition here.

Ok, I am keeping this in my radar! I won't promise it anytime soon, but at some point in my own project, I will have to deal again with connection to cloud storage. I feel doing it with gdrive in this project might profit to others as well. Thanks again!

gdrive is not really mass storage

10Go for free is quite decent I feel :) ah ah but ok, it's not in the To scale...

[martindurant]

10Go for free is quite decent

I meant more in the sense of latency, concurrent connections and throughput. It being free for people not willing to pay the big cloud vendors is a big plus. Be sure to also checkout the fsspec dropbox implementation, which is more complete.

[Coderambling]

Hi @martindurant . Could you give a short update on maturity and status of this project? Should it still be considered experimental? Or stable and mature? Or somewhere in between? Thank you.

[martindurant]

I would say inbetween: it is known to work for certain workflows and specific auth mechanisms, but it is far from tested across all the things that gdrive can do. Testing is hampered by there being no offline testing emulator as there is for GCS.

[Coderambling]

Thanks! I don't suppose there is a list of supported workflows etc. for various filesystems etc.?

To clarify: Do you mean fsspec.gui is known to work for certain workflows and specific auth mechanisms of gdrive specifically?

Are those certain workflows and auth mechanisms listed anywhere by chance?

There are 24 built-in implementations. Are those fully supported in terms of fsspec.gui? Or does the same apply as for the gdrive external package?

24 Built-in: API Reference — fsspec 2024.2.0+1.g510ca5d.dirty documentation

15 Through external packages:

https://filesystem-spec.readthedocs.io/en/latest/api.html#external-implementations

Do people use other GUI-based utilities with fsspec, that are perhaps more popular / de facto standards? Or does every one just use code?

To me it seems like

[martindurant]

fsspec is mainly used in code, and internally by other libraries. The GUI component isn't particularly polished and doesn't have a huge amount of features, but exists as a handy interface and example of the kind of applications that might wrap fsspec. I am not aware of more complete fsspec GUI implementations, but it wouldn't surprise me if there were some; a jupyter-lab side panel, for example, would be an obvious thing for someone to want to make (but not me!).

You should expect all of the builtin implementations to work generally, and all have CI tests that run regularly. The docstrings of each class may give caveats of known situations that aren't covered. Similarly, the majority of the external "known implementations" are stable, particularly the most-used ones: s3fs, gcsfs, adlfs. Again, the README of each project should say something about their status, and each may have different maintainers even when hosted in the fsspec/ github org.

[Coderambling]

Related discussion reference for others: https://github.com/fsspec/gdrivefs/pull/42