"allocation size overflow" error on large files

[cormander]

Large files throw an "allocation size overflow" error on the console when you select a large enough csv file (happening in firefox):

http://localhost:3000/#/import/

react-csv-reader under the hood uses papaparse, so probably related to this issue: https://github.com/mholt/PapaParse/issues/469

The following will create a ~ 288MB large.csv file pretty quick to test this with:

echo "1,2,3,4,5,6,7,8,9" > large.csv; for i in $(seq 1 8); do cat large.csv large.csv large.csv large.csv large.csv large.csv large.csv large.csv > large.csv2; mv large.csv2 large.csv; done

A quick google query pull this up: https://stackoverflow.com/questions/31722808/how-to-papa-unparse-a-huge-json-object

It probably boils down to we have to be careful how we manage processing of a large file, and use byte streams instead of full copies in memory.