Improving Uploads of Big Files via Web Browsers

[lidel]

This a a meta-issue tracking known issues and ongoing improvements related to uploads of big (1GB+) files in web browser contexts.

Known Problems with HTTP API

Invalid progress reporting:

• [ ] Streaming versions of ipfs.files.add buffer entire data in memory before doing HTTP POST
  • upstream: https://github.com/ipfs/js-ipfs-api/issues/842
• [ ] Chromium crash when sending <200MB over js-ipfs-api
  • IN PROGRESS waiting for release with upstream fix to stream-http: https://github.com/jhiesey/stream-http/pull/93
  • related issues
    • crash when dropping a big file (https://github.com/ipfs-shipyard/ipfs-companion/issues/464)
    • Uploading big files crush page in Chrome (https://github.com/ipfs/js-ipfs-api/issues/654)
    • upload-file-via-browser can only upload file size less than 165MB (https://github.com/ipfs/js-ipfs-api/issues/788)
• [ ] Silent crash in Firefox when Buffer max size is breached:
  • upstream issue: https://github.com/ipfs-shipyard/ipfs-companion/issues/464
    • IN PROGRESS waiting for release with upstream fix to stream-http: https://github.com/jhiesey/stream-http/pull/93
• [ ] Silent error while files.add over HTTP API (http: invalid Read on closed Body)
  • go-ipfs: https://github.com/ipfs/go-ipfs/issues/5168 (should land in go-ipfs v0.4.19 - https://github.com/ipfs/go-ipfs/issues/5168#issuecomment-446804558)
    • IN PROGRESS https://github.com/ipfs/go-ipfs-cmds/pull/116
  • js-ipfs-api: https://github.com/ipfs/js-ipfs-api/issues/797

Ongoing Work

• Fix to stream-http is ready, waiting to be released: https://github.com/jhiesey/stream-http/pull/93
• Add support for chunked uploads to ipfs.add (https://github.com/ipfs/js-ipfs-api/pull/851 | https://github.com/ipfs/js-ipfs/pull/1540 | go-ipfs/?)

[olizilla]

This looks worth tracking, as it's being in an origin trial in chromium and has support from firefox, edge, and safari

Fetch upload streaming lets web developers make a fetch with a ReadableStream body.

– https://www.chromestatus.com/feature/5274139738767360

The linked design doc is great https://docs.google.com/document/d/e/2PACX-1vQ26giyHPPhxrgVorNtkVSsJPOWecc6c6wHFLFbbl27r0BIBEgRwahh2b37Uk7HfXvZoW1Cu_ed-bRm/pub