hail icon indicating copy to clipboard operation
hail copied to clipboard

hailtop.fs `ReadableStream` does not support seeking on streams from blob storage

Open daniel-goldstein opened this issue 1 year ago • 1 comments

What happened?

This code block should work:

import hailtop.fs as hfs

with hfs.open('gs://foo/bar.txt') as f:
    f.seek(2)
    print(f.read())

But this fails with an OSError because the ReadableStreams that we produce from blob storage are not seekable. We normally stream through the whole contents of the blob with one GET, so a seek implementation should probably discern based on buffer size whether to skip bytes in the client's buffer or drop the current stream, update a client-side position, and initiate a new request on the next read.

Version

0.2.116

Relevant log output

No response

daniel-goldstein avatar May 15 '23 14:05 daniel-goldstein