Why is file upload so incredibly slow?

[alionthego]

I am uploading files that are about 300-400kb in size. The average upload time for a single file is 30 seconds which is unacceptable. How can I improve this? I'm suing the following code from your example to upload and have a fast stable wifi connection.

let dataString = "My Data"
let data = dataString.data(using: .utf8)!  
let storageOperation = Amplify.Storage.uploadData(key: "ExampleKey", data: data)
let progressSink = storageOperation.progressPublisher.sink { progress in print("Progress: \(progress)") }
let resultSink = storageOperation.resultPublisher.sink {
    if case let .failure(storageError) = $0 {
        print("Failed: \(storageError.errorDescription). \(storageError.recoverySuggestion)")
    }
}
receiveValue: { data in
    print("Completed: \(data)")
}

[royjit]

Are you seeing 30 seconds for all file uploads or just the first try? The first try might include different api calls to fetch AWS credentials which might take a few seconds more. The subsequent upload calls should be faster.

[alionthego]

I'm seeing a very long period for the first try and then somewhat long periods for subsequent uploads. Even not being the first upload the time it takes is really unacceptable. The same data count size to lambda or api upload is much much faster.

[alionthego]

Not sure how these upload times to S3 are acceptable. Files just 2-3 MB take over 30 seconds. Unacceptable.

[brennanMKE]

@alionthego When this is happening can you count up the number of active threads?

We have been working on some performance improvements and one was just merged which could help with your issue. If other work is in progress and it is causing threads to be blocked it could explain why this operation is delayed. See the issues mentioned at the bottom of the PR below. This update will be included in the next release.

#3872

[diegocstn]

@alionthego we've released the performance improvements of #3872 with Amplify v1.18.1. Let us know if it helps with your issue.

[alionthego]

I am using latest sdk and the s3 file upload is still unacceptably slow. it takes about 30 seconds to upload 400 kb.

[sagarjoshi]

same issue, upload is slow

[orlaqp]

I am seeing the same issue :-| . Here is the code I am using:

import { Storage } from 'aws-amplify';
import { launchImageLibrary, MediaType } from 'react-native-image-picker';

 ...

export const uploadAsset = async (mediaType: MediaType, keyPrefix: string): Promise<UploadResponse | null> => {
    try {
        const res = await launchImageLibrary({
            mediaType: mediaType || 'photo',
            selectionLimit: 1,
        });
        const asset = res.assets?.at(0);

        if (!asset?.uri) return null;
        console.log('asset', asset);

        const response = await fetch(asset?.uri);
        console.log('fetch response', response);

        const blob = await response.blob();
        console.log('blob');

        const key = `${keyPrefix}/${asset?.fileName}`;
        const putRes = await Storage.put(key, blob);
        console.log('put response', putRes);
        console.log(putRes.key);

        // const signedUrl = await Storage.get(key, { download: false });
        // console.log('Signed URL', signedUrl);
        return { key, signedUrl: asset.uri };
    } catch (error) {
        console.error('error', error);
        return null;
    }
};

[harsh62]

@orlaqp Are you using amplify for iOS or JS? From you code snippet, it looks like you are using JS. I would suggest you to open an issue in the JS. https://github.com/aws-amplify/amplify-js/issues

[staticVoidMan]

Same issue. Upload is slow!

Actual Example (~1mb file took ~39seconds):

func uploadData(data: Data, key: String) {
    let startDate = Date()
    print(data) // 991691 bytes

    let options = StorageUploadDataRequest.Options(accessLevel: .private)

    Amplify.Storage.uploadData(key: key, data: data, options: options) { result in
        let endDate = Date().timeIntervalSince(startDate)
        print(endDate) // 38.75
    }
}

I ran a few manual tests on the same file size and the fastest upload I got was 3seconds and the slowest I got was 160seconds! But in most cases it's lands between the 30-50seconds window.

[atierian]

Hey @staticVoidMan, thanks for this info. A couple of questions:

• Which version of Amplify are you using?
• What region are you using for Storage, and are you geographically located in/near that region? You can find the region in your amplifyconfiguration.json under storage.plugins.awsS3StoragePlugin.region.

[staticVoidMan]

@atierian Hey! thanks for reverting. We are using Amplify via SPM, and the version we are at is 1.25.0.

As for the amplifyconfiguration.json, the region is us-west-2 but we would be located us-east. I first did doubt the region but our Android counterpart team is using the same us-west-2 but not facing any upload speed issues. Furthemore, our QA team have also reported this under same network conditions.

[anuragsingla123]

@atierian I am facing the same issue as well. I am using the latest Amplify version 1.27.1 for iOS but it still takes 2-3 minutes in uploading a small file. The S3 bucket is currently in us-east-1 region.

[atierian]

Thanks for letting us know. Slow uploads / transfers is actively being investigated. We'll follow up on this issue with any updates.

[dhaval-dobariya]

I am facing same issue, any solution is there?

[jcjimenez]

@alionthego @anuragsingla123 @dhaval-dobariya @sagarjoshi @staticVoidMan I'm picking up this investigation but haven't been able to reproduce this on my end (including using Apple's Network Link Conditioner). Would you please contact me directly to see if we can have a call to diagnose? Thanks.

[alionthego]

Try a request from a different region. Such as servers in Virginia and client in Hong Kong

[jcjimenez]

Just writing to update you all on progress: I do see very slow upload times (about 2 minutes for a 1.1MB file) being in Texas and uploading to AWS regions such as Mumbai. I'm taking a deeper dive to figure out if there is a root cause (in the library or SDK) apart from the physical distance.

[jcjimenez]

I've found an imperfect work-around using AWSS3StoragePlugin's escape hatch to direct S3 interactions but I believe will find a root cause in a day or two. Stay tuned...

[vish7]

Hi, I am facing the same issue. I am using Amplify 1.28.3 version. 11 MB video file is taking 6 -7 minutes to upload. But in AWS SDK video upload was working fine with AWSS3TransferUtility class. Code:

let fileNameKey = "testing_upload_video_3.mov" let filePath = Bundle.main.url(forResource: "testing_upload_video_2", withExtension: "MOV")

    guard let filename = filePath
    else { return }

    let storageOperation = Amplify.Storage.uploadFile(
        key: fileNameKey,
        local: filename,
        progressListener: { progress in
            print("Progress: \(progress)")
        }, resultListener: { event in
            switch event {
            case let .success(data):
                print("Completed: \(data)")
            case let .failure(storageError):
                print("Failed: \(storageError.errorDescription). \(storageError.recoverySuggestion)")
            }
        }
    )

[jcjimenez]

Thank you for your patience, I've submitted the pull request above (for amplify-swift version 2) which the team will discuss.

[nivritgupta]

is there any solutions for this problem , upload time is incredibly slow for me as well , 3 MB file size takes more then 8 to 9 minutes time for upload. 😔😔

[OldChicken]

is there any solutions for this problem , upload time is incredibly slow for me as well , 3 MB file size takes more then 8 to 9 minutes time for upload. 😔😔 + 1

[babinowich]

We have a use case with large, 50-100MB uploads which can take between 0:00:15 and 1:55:00 on gigabit connections. Even worse, we get no status back as far as the progress in fractionCompleted reliably. It can take 5-10 minutes before the user gets notification at all of any progress, forcing us to fake a 1% increment of progress to keep the user on the page.

Therefore we see two core issues with Amplify swift:

1. Extremely unreliable upload speeds with the same file on the same internet connection on the same device in the same hour
2. The above would be less painful if we had reliable progress updates, but we see unusable upload progress for consumers

At this point, we may be forced to consider alternative storage solutions that are responsive and attentive to known issues that have been stagnating for 5 months.

Code:

static func uploadPreSignedFile(key: String, fileUrl: URL) -> StorageUploadFileTask {
    let key = "uploads/\(key).zip"
    return Amplify.Storage.uploadFile(key: key, local: fileUrl, options: .init(
        contentType: "application/zip"
    ))
}


let uploadTask = StorageHelper.uploadPreSignedFile(key: key, fileUrl: fileUrl)
for await progress in await uploadTask.progress {
    self.viewProgress = .uploadS3(progress.fractionCompleted)
    self.regions[index].progress = self.viewProgress?.getProgress()

    // fraction of upload as we scale upload to progress bar from 10 to 99%
    if progress.fractionCompleted >= 1.0 {
        let value = try await uploadTask.value
        logger.info("Upload complete, value: \(value)")
        fileUrl.stopAccessingSecurityScopedResource()
        self.onUploadComplete(key: scanKey, path: value, index: index)
    }
}

[jcjimenez]

Hi, we have work in progress to make this better but it has taken longer than I anticipated. However, here is a workaround you can use to speed things up:

1. Download PresignedUrlGenerator somewhere into your application/library. This performs the subset of the work used by Amplify in order to prepare for upload.
2. Use by entering something like the example below:

let generator = PresignedUrlGenerator(
    region: "us-east-1",
    bucket: mys3BucketName
)
let url = try await generator.presignedURL(
    for: myKey,
    accessLevel: .private
)
var request = URLRequest(url: url)
request.httpMethod = "PUT"

let session = URLSession(configuration: URLSessionConfiguration.ephemeral)
let task = session.uploadTask(with: request,
                              fromFile: fileURL)
task.delegate = self /* You'll need to make sure to conform to URLSessionTaskDelegate*/
task.resume()

You are likely using amplifyconfig.json in your application, and if that is the case, you should be able to get the relevant region and bucket by doing something like the following:

1. Download AmplifyConfig somewhere into your application/library.
2. Read your amplifyconfig.json file with something like:

let amplifyConfig = try AmplifyConfig.load(from: .main)
let generator = PresignedUrlGenerator(
    region: amplifyConfig.storage.plugins.awsS3StoragePlugin.region,
    bucket: amplifyConfig.storage.plugins.awsS3StoragePlugin.bucket
)

[zachrattner]

Thank you @jcjimenez - do you know if this supports multipart POST uploads? Would it work for 50-100MB uploads?

[ruisebas]

@zachrattner the provided PresignedUrlGenerator only supports the PutObject API, but it should be more than enough for 50-100MB uploads as the API supports up to 5 GB.

[jcjimenez]

@zachrattner I hope this workaround did the trick! For reference, it is subject to the limits posted here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html

[babinowich]

@jcjimenez thank you for the code, it worked for @zachrattner and i in that we now see quicker and more reliable responses regarding progress.

[atierian]

Hey folks, quick update. We've resolved this issue in recent releases. Amplify.Storage.uploadData(key:data:) and Amplify.Storage.uploadFile(key:local:) still use a background session, however when the app is in the foreground the upload speed is dramatically faster. See https://github.com/aws-amplify/amplify-swift/pull/2925 for some improvement results.

Here are the versions where this fix is included: Amplify Swift v2: https://github.com/aws-amplify/amplify-swift/releases/tag/2.9.1 Amplify iOS v1: https://github.com/aws-amplify/amplify-swift/releases/tag/1.30.0

*Note: If you're using v1, make sure that the AWS SDK for iOS version used is at least 2.31.1. If it's not, you'll need to take the following steps depending on which package manager you're using: SPM: Xcode > File > Packages > Update to Latest Package Versions CocoaPods: Delete you're Podfile.lock (if you currently have one) + pod install --repo-update