Better Chunk loading handling to avoid HTTP 416 (Range Not Satisfiable) Error

[t83714]

Fixed: #831

When request content in chunks, PapaParse may trigger HTTP 416 (Range Not Satisfiable) Error under the following two situation:

• we always calculate end as: var end = this._start + this._config.chunkSize - 1; see here. This likely always make the last chunk request to request a range that is beyond the actual file size (unless the actual last chunk size is just equal to the this._config.chunkSize.) Many servers will response HTTP 416 error for this.
  • Solution: use Content-Range header to retrieve the actual file size and use actual content size - 1 as the end for the last chunk
• As we often don't know the actual size of the target file before we request it, it's possible the this._config.chunkSize is configured to a size that is larger than the file size. e.g. the file is only 100 bytes and this._config.chunkSize is configured to 64KB. When this happens, PapaParse will always fail on the first chunk request due to HTTP 416 (Range Not Satisfiable) Error
  • Solution: we should retry (only once) on the first chunk request without Range header on 416 error

[RoelRoel]

I really need this. Why is it not merged?

[pokoli]

@RoelRoel Nobody reviewed the code, will you like to do so? Also How can I ensure that it works? Do we have some way to test it?

[t83714]

@pokoli Will nock based test cases be sufficient to prove its correctness? If they are sufficient and @RoelRoel or anyone else can help with the review, I am happy to add test cases to the PR.

[sztupy]

This bug effectively makes PapaParse unusuable as a stream CSV loader on some webservers, so it would be good to have this merged with a good regression test to avoid this breaking again in the future. I am happy to lend a hand if needed

[jsmolina]

this issue should be merged asap

[RoelRoel]

In our app which runs locally in a browser we now download the file first so the browser has it in its cache so you won't get the 416 error. But it would be nice to be able to remove this workaround. At the moment I unfortunately cannot find time to add tests to this library, maybe later when I have less work to do.