amplify-swift
amplify-swift copied to clipboard
aws-amplify
Amplify Storage Progress either returns 0 or 100 nothing in between
Describe the bug
let options = StorageUploadDataRequest.Options(accessLevel: .protected,
targetIdentityId: identityID)
let uploadOperationTask = Amplify.Storage.uploadData(key: filename,
data: data,
options: options,
progressListener: { (progress) in
progressHandler?(progress)
}, resultListener: { (result) in
switch result {
case Result.success :
completion(nil)
case let Result.failure(error):
completion(NetworkingError.apiError(description: error.errorDescription))
print(error)
}
})
return UploadTask(task: uploadOperationTask)
If the above is followed , and if the following contents of the progress object is printed
print("progress is ",progress.totalUnitCount,progress.fileCompletedCount, progress.completedUnitCount)
the output is
progress is 309536 nil 309536 progress is 309536 nil 309536
which means total progress calculated is always 100 , nothing in between
Steps To Reproduce
1. Upload a file
2. Print contents of the Progress object returned as shown above
Expected behavior
Progress object 's completed unit count should vary from 0 to totalUnitCount
Amplify Framework Version
1.18.1
Amplify Categories
Storage
Dependency manager
Cocoapods
Swift version
5.0
CLI version
7.6.5
Xcode version
13.2.1
Relevant log output
No response
Is this a regression?
No
Regression additional context
No response
Device
iPhone 8 Simulator
iOS Version
iOS 15.1
Specific to simulators
No response
Additional context
No response
Hey @iThink32, thanks for opening this issue.
The progressListener will only be called if the size of the upload necessitates it. In your example, 309kb isn't large enough for that to be the case.
If you tried this with a larger upload, you'd see that the progressListener is called and behaves like you'd expect.
For example, this upload:
let mb = 1_000_000
let size = 2 * mb
let bytes = [UInt32](repeating: 0, count: size).map { _ in arc4random() }
let data = Data(bytes: bytes, count: size)
Amplify.Storage.uploadData(
key: "someimage.png",
data: data,
progressListener: handle,
resultListener: { _ in }
)
func handle(_ progress: Progress) {
print("""
-----
* UploadProgress *
Total Unit Count: \(progress.totalUnitCount)
File Completed Count: \(progress.fileCompletedCount as Any)
Completed Unit Count: \(progress.completedUnitCount)
Fraction Completed: \(progress.fractionCompleted)
Is Finished: \(progress.isFinished)
""")
}
will output:
* UploadProgress *
Total Unit Count: 2000000
File Completed Count: nil
Completed Unit Count: 1048576
Fraction Completed: 0.524288
Is Finished: false
-----
* UploadProgress *
Total Unit Count: 2000000
File Completed Count: nil
Completed Unit Count: 1572864
Fraction Completed: 0.786432
Is Finished: false
-----
* UploadProgress *
Total Unit Count: 2000000
File Completed Count: nil
Completed Unit Count: 2000000
Fraction Completed: 1.0
Is Finished: true
Please let me know if this clarifies this for you. Thanks!
@atierian ill tell you the practical implication of this.
First off , even if it says 100% it takes 10-15 seconds after showing so to complete.
Second most ppl upload data of size 1-5 mb from phones for normal purposes like profile photo update , status update etc. In such cases waiting for 15 seconds at a progress of 100 seems wrong.
This is not practical for me , im ok with 100 but provided it finishes which is not the case.
Thanks for the further insights. From cursory testing, the progressListener seems to be called with < 1 fractionCompleted values at about 1.05mb regardless of the network connection quality.
Are you seeing the upload take 10-15 seconds in the example you provided above with 309kb?
Uploading that size while simulating a very bad network connection - 1 megabit/s upload and download bandwidth, 10% packet loss, and 500ms delay - results in receiving a Progress where isFinished is true within a few seconds.
yes @atierian the progress shows 100% and it still needs approx 10 seconds more. I have a decent network connection which is approx 10-20 mbps.
The progress only coming back as 100% makes sense considering the size of the upload. Is the 10-20mbps connection you mentioned the upload speed? And are you using a region that is geographically best suited for your location? If not, doing so will most likely get you the biggest speed wins here.
@iThink32 There seems to be something else happening. Make sure to run your app with the Main Thread Checker enabled. When I have observed long delays it was from running code off main which needed to be on main.
The SDK gets updates from the delegate functions for URLSessionTask (this method) which will be run at the discretion of the OS. The progress is updated in response to calls to that method. The delegate method typically is run off the main thread and may be observed off the main thread in your app. At the point you are updating the UI you want to switch to the main thread with code below.
DispatchQueue.main.async {
// update UI
}
What I have found if the UI is updated off the main thread there can be a long delay for it to catch up and you normally won't see any errors.
@atierian you can consider an upload speed of 5-10 mbps. Im using Ohio but it is not my region of operation , could you give me further insights on how i can redirect storage to an appropriate location?
@brennanMKE as far as i know ui updates are on the main thread itself but ill check it again
@brennanMKE all callbacks happen on the main thread, the issue still persists.
@brennanMKE @atierian another example
for the statement:
print("progress is ",progress.totalUnitCount,progress.fileCompletedCount, progress.completedUnitCount)
image size in MB is 11 compression factor is 0.1 image size in MB is 11 compression factor is 0.1 progress is 244929 nil 244929 progress unit count 244929 progress completed 244929 total progress is 100 progress is 244929 nil 244929 progress unit count 244929 progress completed 244929 total progress is 100
A file size of 11 mb starts with 100% and takes 15 seconds post that to finish
All callbacks on main thread.
The upload in your example is 245kb after compression, so it makes sense that you're not getting a < 1 progress update. However, the 15 seconds you're experiencing is definitely too long - an upload of that size should happen very quickly.
We're unable to reproduce this long delay in any of our testing. Can you please provide us with a small example that reproduces this issue (without any confidential data)?
hey this is the code im using to upload files
func uploadFiles(dataModels: [ImageUploadableModel]) {
self.resetData()
self.totalProgressCalculator = TotalProgressCalculator(numberOfItems: dataModels.count)
self.delegate?.percentageUpdateReceived(str: "Uploading 0%")
//cancel prev operations before new ones start
self.stopOperations()
PostLoginServices.fetchAuthCredentials {[weak self] authCredentials in
guard let unwrappedSelf = self,
let unwrappedIdentityID = authCredentials.identityID else{
self?.delegate?.errorDescriptionReceived(str:
DisplayMessageManager.imagesFailedToUpload)
return
}
dataModels.forEach { (model) in
unwrappedSelf.dispatchGroup.enter()
guard let unwrappedSelf = self else{
unwrappedSelf.dispatchGroup.leave()
unwrappedSelf.modelsFailed.append(model)
return
}
let filename = model.identifier
//save instance for state handling
let uploadTask = Services.uploadFileToStorage(data: model.data,
filename: filename,
identityID: unwrappedIdentityID) { (progress) in
guard let totalProgressCalculator = unwrappedSelf.totalProgressCalculator else{
return
}
print("progress is ",progress.totalUnitCount,progress.fileCompletedCount, progress.completedUnitCount)
Task(priority: TaskPriority.high) {
let totalProgress = await totalProgressCalculator.computeTotalProgressUsing(identifier: filename,
progress: progress)
DispatchQueue.main.async {
print("total progress is \(totalProgress)")
unwrappedSelf.delegate?
.percentageUpdateReceived(str: "Uploading \(totalProgress)%")
}
}
} resultHandler: { (error) in
defer {
unwrappedSelf.dispatchGroup.leave()
}
guard let _ = error else{
unwrappedSelf.modelsSucceeded.append(model)
return
}
unwrappedSelf.modelsFailed.append(model)
}
unwrappedSelf.uploadTasks.append(uploadTask)
}
unwrappedSelf.dispatchGroup.notify(queue: DispatchQueue.main) {
guard unwrappedSelf.modelsFailed.isEmpty == false else{
unwrappedSelf.delegate?.credentialReceived(identityID: unwrappedIdentityID)
unwrappedSelf.delegate?.uploadCompleteFor(models: unwrappedSelf.modelsSucceeded)
return
}
DispatchQueue.main.async {
unwrappedSelf.delegate?.credentialReceived(identityID: unwrappedIdentityID)
unwrappedSelf.delegate?.errorDescriptionReceived(str:
DisplayMessageManager.imagesFailedToUpload)
}
}
}
}
Im waiting for the result to come back but unfortunately that takes a long time
Calling Func
public class func uploadFileToStorageUsing(data: Data,
filename:String,
identityID: String,
progressHandler:
Typealias.ProgressCompletionCallBack,
completion:@escaping
Typealias.ErrorCompletionCallBack) -> UploadTaskDelegate {
let options = StorageUploadDataRequest.Options(accessLevel: .protected,
targetIdentityId: identityID)
let uploadOperationTask = Amplify.Storage.uploadData(key: filename,
data: data,
options: options,
progressListener: { (progress) in
progressHandler?(progress)
}, resultListener: { (result) in
switch result {
case Result.success :
completion(nil)
case let Result.failure(error):
completion(NetworkingError.apiError(description: error.errorDescription))
print(error)
}
})
return UploadTask(task: uploadOperationTask)
}
struct ImageUploadableModel {
let data: Data
let identifier: String
}
@atierian Please check if i have done something wrong in the above , else let me know what else can i do to help you debug the issue
Thanks for this example @iThink32. It appears you're using a lot of custom types / methods in your code snippet that make this difficult to effectively reproduce on our end. Let's try this the other way around, if you make an 2mb upload with this basic example, how long does it take?
let mb = 1_000_000
let size = 2 * mb
let bytes = [UInt32](repeating: 0, count: size).map { _ in arc4random() }
let data = Data(bytes: bytes, count: size)
Amplify.Storage.uploadData(
key: "someimage.png",
data: data,
progressListener: handle,
resultListener: { _ in }
)
func handle(_ progress: Progress) {
print("""
-----
* UploadProgress *
Total Unit Count: \(progress.totalUnitCount)
File Completed Count: \(progress.fileCompletedCount as Any)
Completed Unit Count: \(progress.completedUnitCount)
Fraction Completed: \(progress.fractionCompleted)
Is Finished: \(progress.isFinished)
""")
}
hey @atierian sorry for the long delay in replying , will test this and let you know in a while.
hey @atierian tried the above but this is even worse its taking more than a min to complete the above task.And the same happens, after this log it takes a long time to get the completion handler back
progress is 2000000 nil 1048576
progress is 2000000 nil 1572864
progress is 2000000 nil 2000000
i get the completion handler approx 20 seconds after the progress prints the last statement above
@atierian two things firstly:
- Irrespective of the size of the item being uploaded we must show the progress from 0-100 and not default to 100 in case of smaller files , reason being different network conditions portray different times of completion and an accurate visualisation is the right way forward.
- Due to the above it is hard to say whether the upload is the problem being faced or the completion handler not returning fixing the above should help narrow down the problem.
I've prepared a sample project with 2 versions to observe uploads. The first uses the Amplify library and the other uses Transfer Utility directly from the SDK. I am using the Network Link Conditioner from Apple the simulate network behavior for Edge, 3G and LTE networks. There are delays as expected but I am not able to observe any reason why the apps would be blocking. Please try these apps for yourself. You will need to run setup.sh in the root to clone the repos which are needed and also provide your own configuration to AWS & Amplify to access an S3 bucket.
Let us know if you observe this behavior with these sample projects.
@brennanMKE thanks for this will check and keep you updated in a day or two
hey Brennan, sorry for the very late reply, was badly held up in something.
Ok so tried Amplify results are as follows:
Progress: 0.33 [23B081E7-F8CB-4134-AE0B-DA8C13161902] Progress: 0.50 [23B081E7-F8CB-4134-AE0B-DA8C13161902] Progress: 0.67 [23B081E7-F8CB-4134-AE0B-DA8C13161902] Progress: 1.00 [23B081E7-F8CB-4134-AE0B-DA8C13161902]
There above is for uploading small files
Even though I got a 403 at the end the above result is exactly what I expect. In my case it always returns a progress of 1.00
But if you look at this comment https://github.com/aws-amplify/amplify-ios/issues/1621#issuecomment-1065908806 it always returns 1.0 for me.
My request here is irrespective of the file size please provide a progress of 0-1 and do not default to 1.00 for small files. On a bad network small files take time and Amplify repeatedly always 1.00
hey guys pls consider this request, it would help me a lot and a lot of users who have a similar use case. I really hope this is being developed. @brennanMKE @atierian
Hey @iThink32, the progress reporting is currently limited by what the underlying URLSession reports. But we are looking into this from two different angles, providing accurate progress updates with smaller uploads and improving transfer speed. We'll update you here with any new developments. Thanks again for opening the issue and your patience.
@atierian I've been patient for a few months but now I'm facing issues that are effecting my app.
StorageError: The HTTP response status code is [403].
Recovery suggestion: .
For more information on HTTP status codes, take a look at
https://en.wikipedia.org/wiki/List_of_HTTP_status_codes
when I call download url the above is shown, however on retrying it again it succeeds. Please check the above.
my calling code is
let options = StorageDownloadDataRequest.Options(accessLevel: StorageAccessLevel.protected,
targetIdentityId: identityID)
_ = Amplify.Storage.downloadData(key: path, options: options) { (result) in
switch result {
case let Result.success(data) :
completion(data,nil)
case let Result.failure(error):
completion(nil,NetworkingError.apiError(description: error.errorDescription))
print(error)
}
}
```
This is very frequent off late and is replicable, I've even upgraded to the latest version of S3 and tested but still the same.
@atierian if I have to create a new issue, please check this
https://github.com/aws-amplify/amplify-ios/issues/2228
Hey @iThink32, as commented on the #2228 we've released a fix for this. Thanks for your patience!