Towards a roadmap

[martindurant]

Following a get-together of a few contributors, we have some thoughts on topics that could be part of a mid-term roadmap for fsspec. Here I will list them, with the intention that they be linked to issues discussing each in depth, when there is enough momentum to try to tackle them.

This issue might become a doc page for better visibility. Edits and comments are welcome.

• organisation structure of fsspec
  • possible move to a separate github org with a fully-fleshed identity (coherent docs, logo). Can include repos currently in various places (e.g., dask, intake, dvc, ...)
  • feedback and conversation options for users, better than GH issues. GH discussions maybe the easiest starter.
• test-specific repo
  • regular (e.g., nightly) testing of all implementations
  • a clear path to register new implementations and an expected interface for them to participate
  • dashboard or alert system to surface problems before release
  • obvious place for benchmarking
• push for performance on a number of fronts
  • (optional) threaded operations for non-async filesystem batch operations
  • move to better backend libraries (e.g., ditch paramiko)
  • benchmark-specific cases, ideally with some transparent feedback metric
• directory and path utilities
  • make multiple pluggable dircache implementations, including persistence
  • iter / gen / pagable listings
  • generalised string and maybe Path utilities in concrete location with better testing
  • module of functions operating on URLs, dispatching to filesystems