FileLoader: Allow HTTP Range requests

[takahirox]

Fixed #24485

Update: The proposed API has been updated. See https://github.com/mrdoob/three.js/pull/24580#issuecomment-1247199031

Description

This PR introduces HTTP range requests to FileLoader.

It is helpful to save network usage by downloading only needed part of a file.

Proposed API

const loader = new FileLoader();
loader.setRange(8 /* offset in bytes */, 16 /* length in bytes */); 
loader.load(url, data => { ... });

Changes

• Accept HTTP response status 206 Partial Content
• Add FileLoader.setRange(offsetInBytes, lengthInBytes) method.
  • Range request header can be set even with FileLoader.setRequestHeader() but FileLoader want to know the range because of the following items so having a new clear method would be simpler (by avoiding to use regex for this.requestHeader.Range)
• Take into account the range in the key for Cache (for fetched data cache) and loading (for duplicated requests management).
  • Otherwise, different range requests to the same url can be wrongly handled as the same request.
• Add fallback for servers not supporing range requests.
  • They can respond with 200 and the full body content. If it happens, .slice() for arraybuffer and blob response types, and throw an error for other types because of the complexity to rescue them. Another option may be throw an error regardless of response types and ask users for fallback in their onError callback.

Example use case

glB bundle + glTF LOD extension + progressive loading. First load the lowest levels and then progressively load higher levels on demand.

Currently FileLoader and GLTFLoader don't support HTTP range requests so they load the entire content. With HTTP range request they will be able to partially load files so can save network usage.

Codes

• https://github.com/mrdoob/three.js/compare/dev...takahirox:three.js:RangedRequestGLTFLoader
• https://github.com/takahirox/three-gltf-extensions/blob/main/loaders/MSFT_lod/MSFT_lod.js

Additional context

When previously HTTP range requests support is discussed, (if I understand correctly) there seems to be a chance that content length in onProgress callback can't be computable if servers support gzip Content-Encoding.

But the recent FileLoader already checks if the content length is computable and it's notified to the callback. So, non-computable content length may not be a big deal (?).

https://github.com/mrdoob/three.js/blob/f5e2d49247143b69778a9bbe0077041d5b92c37e/src/loaders/FileLoader.js#L112-L114

[donmccurdy]

Hi and thank you @takahirox!

glB bundle + glTF LOD extension + progressive loading. First load the lowest levels and then progressively load higher levels on demand.

There's a lot I'm not sure about within this workflow...

• if you're using LODs, do you often want separate vertex streams for each LOD? it's also possible to use a single vertex stream and just have alternate indices for higher LODs, MSFT_lod supports this. Makes good sense for maintaining framerate, and uses less bandwidth and memory overall, but doesn't help with progressive loading of course.
• for progressive loading what do you think of using range requests vs. partitioning data into multiple .bin files? e.g.

gltf-transform partition in.glb out.gltf --meshes

We've supported lazy-loading in GLTFLoader for a while, and glTF-Transform has supported partioning data into multiple .bin files for a while, but honestly I don't get the feeling that very many users are making much use of either of these features. So I am a bit worried about adding complexity to GLTFLoader for what feels like a different way of doing the same thing.

I am happy with the idea of THREE.FileLoader supporting range requests. I am still trying to decide about GLTFLoader. There are some other use cases we could think about too, like streaming mip levels (low to high res) of KTX2 textures with HTTP Range Requests, the KTX2 format is designed to allow that. It is just all a tradeoff in terms of complexity and goals of GLTFLoader. 😅

[takahirox]

Hi Don @donmccurdy , thanks for the comments.

I am happy with the idea of THREE.FileLoader supporting range requests. I am still trying to decide about GLTFLoader.

I think it would be better to separate FileLoader HTTP range requests support and GLTFLoader HTTP range requests support discussions. The concerns about GLTFLoader HTTP range requests support don't need to block FileLoader HTTP range requests support. Let's discuss GLTFLoader stuffs in #24506.

Most of your comments look GLTFLoader related. I will reply in #24506.

[mrdoob]

@donmccurdy

I am happy with the idea of THREE.FileLoader supporting range requests. I am still trying to decide about GLTFLoader.

I think @elalish has a few opinions about this.

[sunag]

It is possible too creating an streaming linear LOD for progressive loader. I made this in sea3d for texture and geometry, it is more efficient once it not needed multiples request.

Example: https://sunag.github.io/sea3d/ using one single file, one request, progressive loader:

[elalish]

This looks great, but my interests are in GLTF, so I'll comment in #24506.

[donmccurdy]

Add fallback for servers not supporing range requests. They can respond with 200 and the full body content. If it happens, .slice() for arraybuffer and blob response types, and throw an error for other types because of the complexity to rescue them. Another option may be throw an error regardless of response types and ask users for fallback in their onError callback.

The fallback feels dicey to me. We're creating a new FileLoader for each request, progressive loading will make multiple requests, and caching is disabled by default. This may lead to downloading a full asset multiple times if the server doesn't support range requests.

I'd suggest that HTTP Range requests should be opt-in for downstream classes like GLTFLoader and KTX2Loader, and should fail in FileLoader if the server does not support them, or at least log warnings.

---

I'm also wondering if HTTP Range requests might be better routed through a new Promise-based method, and just not be supported through the load(onLoad, onProgress, onError) signature?

const loader = new FileLoader();

loader.onProgress(() => { ... });

const result = await loader.loadAsync('path/to/file.glb', options);

The options argument above could extend any headers already configured on the loader.

[takahirox]

This may lead to downloading a full asset multiple times if the server doesn't support range requests.

Curious to know if we can expect the browser's cache. Or are the response data distinguished because they are responses to different range headers requests? If browser cache works, might be acceptable. I will check Chromium so far.

should fail in FileLoader if the server does not support them, or at least log warnings.

I'm ok that FileLoader fails for 200 response against range request as I raised as another option because it can avoid additional complexity in FileLoader.

Another option may be throw an error regardless of response types and ask users for fallback in their onError callback.

I'm also wondering if HTTP Range requests might be better routed through a new Promise-based method, and just not be supported through the load(onLoad, onProgress, onError) signature?

I may like this idea. Honestly I don't prefer .setRange() but didn't want to change the .load() API. This idea doesn't break the API.

[takahirox]

I have updated the PR to simplify. New proposed API (no change from the current API).

const loader = new FileLoader();
loader.setRequestHeader({Range: 'bytes=8-23'});
loader.load(url, data => { ... });

I have omitted .setRange() method. I started to think that it may be confusing to users if there are two wayt to set up range, .setRange() and .setRequestHeader(). Providing only one way may be less confusing.

Don's overriding request header idea is interesting. But I want to focus on minimal change in this PR. I hope we can think of better API in another PR if needed.

And I changed to always call onError callback if servers respond with 200 + full body content against range requests because the implementation can be simpler.

Changes

• Accept HTTP response status 206 Partial Content
• Take into account the range request header in the key for Cache (for fetched data cache) and loading (for duplicated requests management).
  • Otherwise, different range requests to the same url can be wrongly handled as the same request.
• Call onError callback if servers respond with 200 and the full body content against range requests because they don't support them.

[LeviPesin]

Call onError callback if servers respond with 200 and the full body content against range requests because they don't support them.

I'm not sure this is the best. I think for such servers the entire response should be cached with key-without-range and then for each range just a portion of response should be returned, like it was before.

[takahirox]

Adding workarounds adds complexity. I don't think HTTP range requests is main stream use case so I don't think FileLoader needs to be complex for non-mainstream use case. It calls onError callback with response so fallback can be done on user end.

[takahirox]

What blocks this PR? Do we need more discussion about fallback?

[takahirox]

I want to add one comment for fallback. FileLoader will call onError callback with response if server responds with 200 for range requests. response has full range content. If users want to avoid duplicated full range request, they can write fallback on their ends, for example caching the full content and return sliced buffer for range requests coming next. This can avoid to make FileLoader more complex than it really needs to be. (Remember that probably range requests is a minor use case. FileLoader doesn't need to be complex for minor use cases.)

loader.setRequestHeader({Range: 'bytes=...'});
loader.load(url, onLoad, onProgress, (error) => {
  const response = error.response;
  if (response && response.status === 200) {
    // fallback on user end.
    // response has full content
  }
});

And I speculate even duplicated full range requests won't be a big deal thanks to browser's disk cache.

If you folks think the name "HttpError" is confusing because 200 is not an error, we may add "RangeRequestError".

[ycw]

But the recent FileLoader already checks if the content length is computable and it's notified to the callback. So, non-computable content length may not be a big deal (?)

@takahirox lengthcomputable maybe false positive because fileloader doesn't handle content-encoding header, if server responds with compressed content, or https://github.com/mrdoob/three.js/issues/24962#issuecomment-1319526224, loaded/total will >1, it can be verified by servers that support compression, ex. githack: http://raw.githack.com/mrdoob/three.js/dev/examples/index.html#webgl_renderer_pathtracer "loading...599%"

[takahirox]

Thanks for the comment. If I understand correctly, the problem is not range requests specific so it won't block this PR, and we can work on resolving it in another PR if possible and needed, isn't it?

[ycw]

@takahirox yes.

[nipunavexev]

Hi, was wondering if there was anything preventing this PR from being merged? We have a use case for this and would love to have it merged :)