filesystem_spec
filesystem_spec copied to clipboard
fsspec
A specification that python filesystems should adhere to.
see e.g. https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.implementations.reference.ReferenceFileSystem 
The `AbstractFileSystem` states for the `cat()` operation: ```python def cat(self, path, recursive=False, on_error="raise", **kwargs): """Fetch (potentially multiple) paths' contents Parameters ---------- recursive: bool If True, assume the path(s) are directories,...
Following a get-together of a few contributors, we have some thoughts on topics that could be part of a mid-term roadmap for fsspec. Here I will list them, with the...
The ```ls``` function is broken and raises an exception if a repo contains a git submodule. Exception is caused by fact that 'type' can be either a 'blob', 'tree' or...
As you well know, write/delete operations are exclusive under Windows. It is quite common to run into `PermissionError`s trying to delete a file that is being scanned (read) by an...
Not even sure if this makes sense. Still thinking through this myself. So maybe this is just a starting point for this discussion. Though this has been coming up in...
See: https://github.com/intake/filesystem_spec/pull/762#issuecomment-923967545
Consider the following methods ```python @staticmethod def _decode_stat(stat, parent_path=None): if S_ISDIR(stat.st_mode): t = "directory" elif S_ISLNK(stat.st_mode): t = "link" else: t = "file" out = { "name": "", "size": stat.st_size,...
I recently noticed that when I developing fsspec, the test step of Github actions were running very slow then locally. After dig into logs, I think a lot of time...
**Description** We were using the `check` and `create` parameters to make sure we have write permission to the bucket before we do a lot of time-consuming work. When using `s3fs`...
