emscripten icon indicating copy to clipboard operation
emscripten copied to clipboard

Fix progress reporting for compressed downloads in fetchRemotePackage

Open vjyanand opened this issue 7 months ago • 3 comments

When a server uses compression (e.g., Brotli (br) or Gzip (gzip)), the Content-Length header in the HTTP response represents the compressed size of the data, not the uncompressed size. The fetchRemotePackage function, however, tracks the uncompressed size of the downloaded data (via value.length of each chunk after decompression). This leads to a discrepancy where the reported totalLoaded (uncompressed) exceeds the Content-Length (compressed), causing inaccurate progress reporting.

The Fetch API does not provide direct access to the compressed stream’s size for each chunk, as decompression is handled transparently. The updated code addresses this by aligning the total and loaded values with the compressed size from Content-Length when the content is compressed. Specifically:

  • The contentEncoding is checked to determine if compression (gzip or br) is used.
  • If the content is not compressed and Content-Length is available, total is set to the Content-Length value. Otherwise, it falls back to packageSize.

This ensures progress reporting reflects the compressed size, preventing totalLoaded from exceeding total.

vjyanand avatar Apr 18 '25 12:04 vjyanand

Can you update the PR description and title now that we have changed the PR content?

sbc100 avatar Apr 23 '25 15:04 sbc100

Do you know why we cannot just use packageSize in all cases? Are there some cases where packageSize is not known?

sbc100 avatar Apr 23 '25 18:04 sbc100

Do you know why we cannot just use packageSize in all cases? Are there some cases where packageSize is not known?

just my guess, packageSize is calculated while doing packaging, one can still change the file with different size on the server without doing packaging. in that case, content-length will be correct value,

vjyanand avatar Apr 23 '25 18:04 vjyanand

It looks like someone else just ran into this in #24529. Any chance you could update this PR and we can land it?

sbc100 avatar Jun 09 '25 18:06 sbc100