fsspec
write/flush asynchronous
More question first: is flushing to gcs asynchronous ? i.e. I like this not to be a blocking operation so that the buffer (and new one ?) gets filled in while the previous block is being written to backend.
If it is a blocking op, request is to not make it ;-) or provide an option to enable/disable.
Our use case is a high-volume high-speed module in flask (switching to aiohttp server) where we continuously stream to GCS during a time window.
Writes append to a buffer, which is pushed to remote when it gets bigger than the configured block size. That push is blocking. It would take work to make async and not all fsspec backends would support it. However, if you use Dask, you can already pass work off to its scheduler, to be executed concurrently.
On October 22, 2019 12:30:07 PM EDT, Yannick Einsweiler [email protected] wrote:
More question first: is flushing to gcs asynchronous ? i.e. I like this not to be a blocking operation so that the buffer (and new one ?) gets filled in while the previous block is being written to backend.
If it is a blocking op, request is to not make it ;-) or provide an option to enable/disable.
Our use case is a high-volume high-speed module in flask (switching to aiohttp server) where we continuously stream to GCS during a time window.
-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/dask/gcsfs/issues/188
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
