django-file-form
django-file-form copied to clipboard
mbraak
Number of concurrent uploading files.
Is there a way to limit the number of concurrent uploading files, for one instance or even across sessions? Right now during stress testing many file upload fails if I try to upload multiple files. In this case I think djangl-file-form should allow retry, even retry automatically if the failure was caused by limited resources such as RAM.
The tus-js-client supports automatic retries. I'll see if I can enable that.
Thanks. I will try it out.
The problem I have now is that I saw some scary error messages such as sys.exit, likely due to the exhaust of RAM or something if I overwhelmed the server with uploads. I am not quite sure if retry, automatic or not, would solve the problem.
If my suspicion was correct, then there needs to be some system-wide control or exception management to make sure that the server will not crash, then resilient retry and/or some queueing/scheduling to manage concurrent uploads.
A final resort, which looks very appealing to me know, is to use a signed post request to upload directly to S3 so that my server does not have to handle any of the upload activity. Right now the server gets all the uploaded files, process, and upload to S3, but it is possible to upload directly to S3, and the server would download, process, and upload. I asked about the use of Uppy in #272 , which was then out of the scope of django-file-form, but given that STORAGE support was removed, it might make sense to consider it again so that django-file-form would upload directly to S3 if boto3 storage is specified. Note that FineUploader is another option that I looked that supports direct S3 upload.
I can confirm that the PR has improved the performance quite a bit, in that I got fewer crashes. However, I could not upload a file that is about 1G in size, which always failed at around 80%, and there were no error message (other than upload failed). Re-uploading did not help and it obviously did not reuse uploaded pieces (always started from beginning).
The setting I have for django-file-form and django is 4G per file.