s3fs icon indicating copy to clipboard operation
s3fs copied to clipboard

Add Seeker interface to file

Open HolzeHan opened this issue 3 years ago • 2 comments

This PR implements the optional seeker interface for file

HolzeHan avatar Aug 10 '22 12:08 HolzeHan

Before I start reviewing, my first question is - why? What's the use case?

Each Seek call essentially makes another call to s3 which costs you money. There are a lot of functions that can take advantage of seeker behind the scenes, potentially making your aws bill higher.

It also introduces some issues, such as you not closing the body while calling seek (if it's opened) thus leaking go routines.

jszwec avatar Aug 10 '22 13:08 jszwec

Currently I am streaming 10GB files from s3 into an application. The file has a streamable format, so it can be read from most positions. It would therefore speed up my processing time, if I could seek in the file. In addition I want to add a wrapper around a fs which allows to stream data using multipart download. For that I need the fs to support the seek operation (every part needs to seek to its starting position and read from there). This would speed up the streaming from around 30MB/s to somewhere around 200MB/s

Leaking go routines is a good point. I'll check again with that in mind

HolzeHan avatar Aug 10 '22 14:08 HolzeHan

LGTM thanks!

jszwec avatar Aug 24 '22 22:08 jszwec