Range Requests

[djherbis]

If you're looking for Range Request support, ~~you can use stream, which supports ReadAt and can therefore be used with io.NewSectionReader to get a io.ReadSeeker for use with ServeContent. I'm planning on modifying fscache to use stream behind the scenes.~~

We're looking into how to support it. fscache is now backed by stream so we have ReaderAt support. The main issue is that http.ServeContent requires an io.Seeker which can seek from the "end" of file. Clearly this is tricky to support since we don't know the EOF index while we are streaming, and don't want to block requests (we'd like to serve everything concurrently that we can).

[djherbis]

@amozoss Is this working for you now? Any more issues? If its working I might tag a new release with this feature fixed.

[amozoss]

Yep, its working. Haven't found any more issues, yet.

[djherbis]

So I've run into a bit of a blocking problem. I started looking to http.ServeContent and it uses some trickery in io.Seeker. It seeks to the "end" using io.Seek(0, os.SEEK_END). This is a problem since we don't know the full length of the file at serving time. Therefore my io.NewSectionReader idea doesn't work well since you have to specify the "length" of the range (I thought you could just put an arbitrarily large number there).

I'm going to go over this a bit more and see if there is a way around this. I really don't want to block clients until the original stream is written. That kinda violates the purpose of fscache.

[lox]

Just noticed this issue, I've been working on a similar concept specifically designed for serving ranges in my httpcache: http://github.com/lox/patchwork

[djherbis]

Hey, yeah it seems like we're solving pretty similar problems.

Are you running into the io.Seek(0, os.SEEK_END) problem as well? We can't really seek to the "end" of the stream without blocking for it to exist. Which kinda hurts us being able to serve ranges as they become available.

One solution is if you are aware of the total size of the stream before its written, you could make it work. But we don't have that.

[lox]

@djherbis patchwork is for where you know the size of the stream you are fetching, I wrote it to specifically cater for my usecase.

[djherbis]

I've added a Size() method to Stream() which allows you to inspect the current stream size, and see if the stream is closed. If it's closed, then the size will be fixed, allowing for range requests to be completed via:

var r stream.Reader
if size, isClosed := r.Size(); isClosed {
    sr := io.NewSectionReader(r, 0, size)
    http.ServeContent(rw, req, r.Name(), modTime, sr)
}
// otherwise just do regular request writing