r2-bucket-uploader icon indicating copy to clipboard operation
r2-bucket-uploader copied to clipboard

Published 20 hours ago • verified publisher icon datopian

Multipart upload fails for large files

Open sorin89 opened this issue 1 year ago • 7 comments

Tested with 5GB and 8GB files. Works fine for a 2GB file.

sorin89 avatar Jan 30 '24 15:01 sorin89

I realized what the problem is. When the file is too large, it gets stuck on creating the arrayBuffer and fileHash.

sorin89 avatar Jan 31 '24 20:01 sorin89

Interesting @sorin89 - computation of file hash would happen locally so I wouldn't expect that being a problem but good to know. I believe we should make it optional.

anuveyatsu avatar Feb 29 '24 11:02 anuveyatsu

I tried using the solution in Nextjs 14, however I'm hvaing 405 error.

eltonvilanculo avatar Mar 06 '24 07:03 eltonvilanculo

I tried using the solution in Nextjs 14, however I'm hvaing 405 error.

Solved, it was related to new update of Nextjs issue

eltonvilanculo avatar Mar 07 '24 13:03 eltonvilanculo

hi @sorin89 have you tried with removing the hashing function? Please let us know if it helped.

also @steveoni would you mind commenting with your latest tests - I know that you've been playing with the multi-part upload recently 😄. We'd be interested in what's the largest size you tried to upload, what is the issues (if any) and did you remove the hashing function?

anuveyatsu avatar Jul 29 '24 06:07 anuveyatsu

hi @sorin89 have you tried with removing the hashing function? Please let us know if it helped.

also @steveoni would you mind commenting with your latest tests - I know that you've been playing with the multi-part upload recently 😄. We'd be interested in what's the largest size you tried to upload, what is the issues (if any) and did you remove the hashing function?

It’s true that for large file , generating the hash string breaks the app. So I tried with a fixed hash string and was able to upload a 10gb file.

So the solution will be to find other alternatives to generate hash string

steveoni avatar Jul 29 '24 08:07 steveoni

@steveoni I would say we don't need to generate file hash at all by default. Hashing is useful if you would like to have:

  1. Versioning so that md5 hash can be used as a signature of the file content.
  2. Save storage, i.e., the same file content wouldn't be stored multiple times.

I think the main purpose of this library is to provide a basic app to be able to upload files from local disk to a bucket.

Could you please open a pull request removing hashing function? We can use file name as a prefix in the bucket.

anuveyatsu avatar Jul 30 '24 11:07 anuveyatsu