Blobs likely are limited to 345MiB in file size

[machawk1]

Not really an issue, but a limitation of https://github.com/eligrey/FileSaver.js .

This could be problematic when appending to existing WARCs or creating WARCs from multiple web pages at once.

[eligrey]

It's a limitation of Chrome, not just FileSaver.js.

[machawk1]

@eligrey , thank you for chiming in. Is there an explanation of why this limit is the case?

[eligrey]

From my testing any blob over exactly 345MiB (not MB) fails to download.

[machawk1]

See also: https://github.com/jimmywarting/StreamSaver.js

[N0taN3rd]

@machawk1 As of Jan 12 2017 Chrome issue 375297 was closed

Except from comment by issue owner during review / ticket closing 107

var b = new Blob([new Uint8Array(700*1024*1024)], {type: 'application/octet-string'}));
var reader = new FileReader();
reader.onload = function(event) {
  console.log("got load event", event);
}
reader.onerror = function(event) {
    console.log("got error", event);
}
reader.readAsArrayBuffer(b);

In-memory quota:

• 2GB if system is x64 and NOT ChromeOS or Android
• Total RAM amount / 5

Disk quota:

• Disk size / 2 if ChromeOS (user partition disk size)
• Disk size / 20 if Android
• Disk size / 10 otherwise

Chrome versions >= 57 include this

Can confirm code snippet works (no errors) in the latest stable Chrome release v59

Seems like 2GB is good maybe see Chrome issue 418743 and MediaStreamRecorder issue 86 reports (Jan 2016) blobs of 954 MB are good.

Must use FileReader and ArrayBuffers + Blob / File

Issue #25 is addressed directly by the Chrome updates mentioned in this comment

[machawk1]

@N0taN3rd Very large payloads, as described in this ticket (and potentially generated per #66), should be a test case in WARCreate. We currently have no test suite integration but need one.

Can you create the beginnings/layout a test suite for the repo (#35)? No need to write the tests just yet but to have the mechanisms in-place to write the tests would be a good stepping stone.

Also, we may want to bump the Chrome version requirements to 57 in the manifest if this is the case.

[N0taN3rd]

@machawk1

Very large payloads, as described in this ticket (and potentially generated per #66), should be a test case in WARCreate. We currently have no test suite integration but need one. Can you create the beginnings/layout a test suite for the repo (#35)? No need to write the tests just yet but to have the mechanisms in-place to write the tests would be a good stepping stone.

I can start it off for sure.

Also, we may want to bump the Chrome version requirements to 57 in the manifest if this is the case.

Agree. The minimum Chrome version bump from would make the implementation of this (and other) functionality which are highly dependant on which version of Chrome you are running less bothersome.

See this breakdown on the browser market share.

The market shares for the lowest three Chrome versions are v31 (0.19%), v38 (0.13%) and v41 (0.41%)

Chrome v58 has largest with 32.14% so the min bump does not appear to be capable of causing dramatic userland incompatibility

[machawk1]

@N0taN3rd Those stats seem believable. At some point, we ought to convert to WebExtensions if, for nothing else, to document the process of going from a Chrome Extension to Web Extensions and any issues we run into. Prior to that, however, it would extremely useful to have a means of evaluating correctness or at least consistent wrongness (test suite per above). Any work you do putting that in-place will help the project progress.