What is the best way to upload large file (video)?

[ngocketit]

Hi,

I'm using Storage.put to upload video files to S3 but the app crashes if file size is large. Here is the code:

const fileData = await RNFetchBlob.fs.readFile(videoUri, 'base64')
const buffer = new Buffer(fileData, 'base64')
Storage.put(fileName, buffer, ...)

Does amplify support multi-part upload yet? If not, what's the workaround?

Thanks!

[elorzafe]

@ngocketit We have a pending PR still in review to support multi-part upload. How large are the files you are trying to upload?

[ngocketit]

@elorzafe We're building a video application and users are allowed to upload videos of maximum 1 minute. So I'd say around 100-150MB. I found a workaround using signed URL but it'd be good if Amplify supports this.

[bilal-korir]

In the documentation, they mention using readStream instead for large file

rn-fetch-blob

If you test it and it works, please share your experience

[snooplsm]

rn-fetch-blob doesn't really work any more with newer react native and expo. I'm trying to use a blob but it seems like the blob uploads a corrupt file. doing something like this:

fetch(file.uri).then(res=>res.blob()).then(blob=>Storage.put('foo',blob)) and everything works fine but the file is like 4x larger than it should be on the bucket. Will there be any examples of uploading real files instead of just text files?

[kkushimoto]

@elorzafe Has there been any progress regarding this issue?

[snooplsm]

I believe there is a bug on react native so you need to manually create the blob this way:

urlToBlob(url) .then(blob=> { return Storage.put("yourkey.m4v", blob, { contentType: contentType }); });

const urlToBlob = url => new Promise((resolve, reject) => { const xhr = new XMLHttpRequest(); xhr.onerror = reject; xhr.onreadystatechange = () => { if (xhr.readyState === 4) { resolve(xhr.response); } }; xhr.open("GET", url); xhr.responseType = "blob"; // convert type xhr.send(); });

This has worked for me.

[EnzoDomingues]

@snooplsm didn't work with me

[EnzoDomingues]

const blob = await new Promise((resolve, reject) => { const xhr = new XMLHttpRequest(); xhr.onload = function () { resolve(xhr.response); }; xhr.onerror = function () { reject(new TypeError("Network request failed")); }; xhr.responseType = "blob"; xhr.open("GET", this.video, true); xhr.send(null); });

i found this solution ˆˆ

[dylan-westbury]

In react native I am able to upload videos like so, as mentioned in the documentation. However the app freeze for a moment and the upload is quite slow.

Has anyone implemented a faster way to upload larger files, like videos?

uploadToStorage = async pathToVideoFile => {
  try {
    const response = await fetch(pathToVideoFile)

    const blob = await response.blob()

    Storage.put('key.mov', blob, {
      contentType: 'image/mov',
    })
  } catch (err) {
    console.log(err)
  }
}

[matamicen]

@ngocketit Does the pre signed URL improves the upload time to S3 comparing with Storage.put ?

@elorzafe Storage.put is taking so long time for files just around 10mb, the same file is 4x faster uploading from the S3 console, do you think the pre signed url is the only way to improve this until Amplify improves the Storage.put performance?

Thanks both of you :)

Matt.

[dorontal]

@elorzafe Any app that handles large file uploads to the cloud must also be able to pause the download or resume it and it must be able to get progress reports on how much has been downloaded thus far - without those features existing on large file uploads, the user interface would be extremely lacking.

So it is not enough to just implement Storage.put() properly for large files.

I noticed that the functionality needed is already present in the AWS Javascript SDK, i.e. here: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html

It has the following functions:

• CreateMultipartUpload
• UploadPart
• CompleteMultipartUpload
• ListParts
• ListMultipartUploads
• AbortMultipartUpload

This is perfect - exactly what we need to implement multipart uploads. Now here's my question:

Could we easily use the AWS SDK for multipart uploads together with AWS Amplify? If so, how? Is there anything special we need to do or any recommended way for using the SDK while also (mainly) using Amplify? How would you go about implementing using the Javascript SDK together with Amplify? What is the recommended way?

Thanks!

[dorontal]

Actualy, found this -- which explains how to use aws-sdk together with Amplify. This seems to be the solution to my point above: https://github.com/aws-amplify/amplify-js/issues/1604

[illiteratewriter]

@dorontal can you share code on how you passed the file to aws-sdk?

[dorontal]

@illiteratewriter I cannot share code because I have not written it yet. But if you follow the two links I shared above, they clearly outline how to do this. The first link lists and documents the functions available for you in the SDK for multipart uploads: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html and the second link has code in it showing how to use the SDK with Amplify: #1604

[walvekarnikhil]

Storage.put supports Managed upload, splitting the file into chunks and uploading into batches. But the problem here is, to call Storage.put we have to pass the blog, which means we have read the full file. Reading full file causes out of memory issues and the app crashes. Here is one solution which I built & tried. https://dev.to/walvekarnikhil/reactnative-large-file-upload-using-amplify-s3-17ho

[viprocket1]

Hey everyone, is this issue resolved yet , im facing the similar problem. my file size is 75MB. My app closes when i click on upload. Thank you.

[jamesaucode]

@dylan-westbury @viprocket1 Seems to be related to https://github.com/aws-amplify/amplify-js/issues/6419, its a bug we introduced for a work around we implemented for working with Axios and React Native.

[jamesaucode]

#8336 might fix some of your issues regarding React Native. Large file uploads will no longer freeze the UI or throw Out of Memory errors

[jamesaucode]

https://docs.amplify.aws/lib/storage/upload/q/platform/js/#pause-and-resume-uploads For larger files we now recommend using resumable upload

[majirosstefan]

@jamesaucode I am uploading videos / files / zips on Android, using AWS Amplify aws-amplify": "^4.3.11", and "aws-amplify-react-native": "^6.0.2".

Each of them has over 100MB. Fetch() call, which is responsible to fetch whole file content into memory, is breaking part of the whole process that results into

TypeError: Network request failed
    at fetch.umd.js:535
    at JSTimers.js:235
    at _callTimer (JSTimers.js:130)
    at Object.callTimers (JSTimers.js:383)
    at MessageQueue.__callFunction (MessageQueue.js:416)
    at MessageQueue.js:109
    at MessageQueue.__guard (MessageQueue.js:364)
    at MessageQueue.callFunctionReturnFlushedQueue (MessageQueue.js:108)
    at debuggerWorker.aca173c4.js:4

Please, consider reopening this issue, as it specifically deals with uploading large videos and I think, that although the current solution is better than before as it no longer freezes the UI, it's still far from optimal one - I would expect that Storage.put would be able to upload several hundreds of MBs into S3 even when using React Native on Android (Android devices are in my eyes less powerful than e.g. something from Apple).

[stocaaro]

Related to #9736

[g-popovic]

Couldn't one use the uploadData method instead of Storage.put? It seems to have automatic multi-part uploading for files exceeding a certain size