csvbase
csvbase copied to clipboard
Uploading a large csv file isn't fast enough
Description
The aspiration is to allow uploaded csv files to be up to 100MB in size and for the general request timeout to be 30 seconds.
That means that csv files have to go into csvbase at least 3.4mbps. Currently we only achieve just over 2 mbps.
Steps to reproduce
- Upload a 100mb file to https://csvbase.com/new-table/upload-file
Expected result
302 redirect to first page of the new table
Actual result
500 error caused by gunicorn timeout
Additional details
We should log the rate after each upload.
We should be transparent on the upload page that we accept files up to 100mb in size.
To be lenient, we should accept 104,857,600 bytes plus some margin to allow for people who have a "100mb" file and then add a header.