hail
hail copied to clipboard
hailtop.fs `ReadableStream` does not support seeking on streams from blob storage
What happened?
This code block should work:
import hailtop.fs as hfs
with hfs.open('gs://foo/bar.txt') as f:
f.seek(2)
print(f.read())
But this fails with an OSError
because the ReadableStream
s that we produce from blob storage are not seekable. We normally stream through the whole contents of the blob with one GET, so a seek implementation should probably discern based on buffer size whether to skip bytes in the client's buffer or drop the current stream, update a client-side position, and initiate a new request on the next read.
Version
0.2.116
Relevant log output
No response
I think this is pretty critical to making the Hail FSes a complete solution for scientists.