filesystem_spec icon indicating copy to clipboard operation
filesystem_spec copied to clipboard

Integration with pyodide-http

Open martindurant opened this issue 3 years ago • 0 comments

https://github.com/koenvo/pyodide-http provides patches to requests and urllib, to enable python stack HTTP calls in the browser. Since several fsspec backends (http, s3, gcs. azure) depend on HTTP, this raises the possibility of fsspec and pydata IO in the browser.

Principle problem:

  • fsspec's backends are async-behind-sync, and use a dedicated IO thread and loop. Perhaps this is analogous to the way pyodide-http uses a webworker. ( https://github.com/koenvo/pyodide-http/issues/11 ). These backends can also be used in async mode, when called from async code, but without file-like objects (since there is no async standard class to duck-type to)
  • CORS (does the s3 API service, for instance, allow global CORS? I can only find info on how to set CORS for the HTTP endpoints of some public bucket)

Alternatively, sync versions of these backends would already be useful, and some existing remote backends are sync and using requests (github, dropbox?)

cc @koenvo @philippjfr

martindurant avatar Oct 10 '22 00:10 martindurant