Unable to upload to S3 - getting 403 AccessDenied errors when uploading to storage

[dorontal]

Description

After setting up S3 storage via the CLI for Auth users to have all CRUD operations, and trying from the app source code to upload a file (tried with all three StorageAccessLevel: .private .protected or .guest and the same issue shows up with any of these), the upload fails with this exception:

StorageException(message: Something went wrong with your AWS S3 Storage upload file operation, recoverySuggestion: See attached exception for more information and suggestions, underlyingException: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: XEGAQR32M4SQ81CT), S3 Extended Request ID: U/okTHe6oLTXpnU1479hpj7CHsL0F+LU9w0M+MkNP+2Hc1/WqFZnocvemmbc5p44kErCGRCXELI=)

The Flutter client app code that caused this exception follows.. The error is raised in the call to Amplify.storage.uploadFile():

  Future<void> uploadProfileThumbnail(File thumbnailFile) async {
    // NOTE: the two lines below print some non zero length so we know that
    // the file is not empty - it is here for debugging  only, remove it
    final len = await thumbnailFile.length();
    dev.log('uploadProfileThumbnail(${thumbnailFile.path}) - $len');

    // Set the access level to `protected` for the current user
    // NOTE: A user must be logged in through Cognito Auth for this to work.
    // NOTE: A user is always logged in as Cognito Auth when we run this
    // NOTE: tried all three StorageAccessLevel -- here -- all three fail the same way
    // NOTE: also tried to give the correct content type which did not fix the 403 access denied error
    final uploadOptions = S3UploadFileOptions(
      accessLevel: StorageAccessLevel.protected,
    );

    // Upload the file to S3 with protected access
    try {
      final key = '${_currentUser!.id}.png';
      dev.log('key: $key');
      final UploadFileResult result = await Amplify.Storage.uploadFile(
          local: thumbnailFile,
          key: key,
          options: uploadOptions,
          onProgress: (progress) {
            dev.log('Fraction completed: ${progress.getFractionCompleted()}');
          });
      dev.log('Successfully uploaded file: ${result.key}');
    } on StorageException catch (e) {
      dev.log('Error uploading file: $e');
    }
  }

I followed all the documentation here to set up S3 storage uploads. If I've missed something, please holler.

Also: Checked the CloudWatch logs and they have nothing in them. There is an S3 trigger set up for this project. The S3 trigger works just fine when I run it in a Javascript version of this client app so I know it is not causing the problem here. Moreover, the S3 logs that are seen in CloudWatch do not show up at all in this Flutter version of the app, which implies that the S3 trigger never got invoked - concluding that the access denied error was at the level of the request for upload, before the S3 trigger had a chance to execute.

Categories

• [ ] Analytics
• [ ] API (REST)
• [ ] API (GraphQL)
• [X] Auth
• [ ] Authenticator
• [ ] DataStore
• [X] Storage

Steps to Reproduce

1. Follow the CLI setup to allow Auth users full CRUD permissions for S3 storage while giving only read access for guests (see CLI transcript and configuration file below)
2. Use code like the function uploadProfileThumbnail() above that is causing the exception, to upload a simple file - either as guest, protected or public - it doesn't seem to matter. NOTE: make sure a user is signed in before calling this function

Here is the storage setup via the CLI:

spawn amplify add storage
? Select from one of the below mentioned services: Content (Images, audio, video, etc.)
✔ Provide a friendly name for your resource that will be used to label this category in the project: · s3
✔ Provide bucket name: · userdata
✔ Restrict access by? · Auth/Guest Users
✔ Who should have access: · Auth and guest users

✔ Who should have access: · Auth and guest users
✔ What kind of access do you want for Authenticated users? · create/update, read, delete
✔ What kind of access do you want for Guest users? · read
✔ Do you want to add a Lambda Trigger for your S3 Bucket? (y/N) · yes


✔ Select from the following options · Create a new function

✔ Select from the following options · Create a new function
✅ Successfully added resource S3Triggerf2792edb locally
✔ Do you want to edit the local S3Triggerf2792edb lambda function now? (y/N) · no

✅ Successfully added resource s3 locally

Screenshots

No response

Platforms

• [ ] iOS
• [X] Android

Android Device/Emulator API Level

API 29

Environment

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.0.4, on Debian GNU/Linux 11 (bullseye) 5.10.0-15-amd64, locale en_US.UTF-8)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.1.0-rc1)
[✓] Chrome - develop for the web
[✓] Linux toolchain - develop for Linux desktop
[✓] Android Studio (version 2021.1)
[✓] VS Code (version 1.68.1)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

Dependencies

Dart SDK 2.17.5
Flutter SDK 3.0.4
tracktunes_app_aws 1.0.0+1

dependencies:
- amplify_api 0.6.1 [amplify_api_android amplify_api_ios amplify_core amplify_flutter aws_common collection flutter meta plugin_platform_interface]
- amplify_auth_cognito 0.6.1 [amplify_auth_cognito_android amplify_auth_cognito_ios amplify_core aws_common collection flutter meta plugin_platform_interface]
- amplify_datastore 0.6.1 [flutter amplify_datastore_plugin_interface amplify_core plugin_platform_interface meta collection async]
- amplify_flutter 0.6.1 [amplify_core amplify_datastore_plugin_interface amplify_flutter_android amplify_flutter_ios aws_common collection flutter meta plugin_platform_interface]
- amplify_storage_s3 0.6.1 [amplify_storage_s3_android amplify_storage_s3_ios amplify_core aws_common flutter meta plugin_platform_interface]
- cupertino_icons 1.0.5
- flutter 0.0.0 [characters collection material_color_utilities meta vector_math sky_engine]
- flutter_hooks 0.18.5+1 [flutter]
- google_fonts 3.0.1 [flutter http path_provider crypto]
- hooks_riverpod 1.0.4 [collection flutter flutter_hooks flutter_riverpod riverpod state_notifier]
- image 3.2.0 [archive meta xml]
- image_picker 0.8.5+3 [flutter image_picker_android image_picker_for_web image_picker_ios image_picker_platform_interface]
- path_provider 2.0.11 [flutter path_provider_android path_provider_ios path_provider_linux path_provider_macos path_provider_platform_interface path_provider_windows]
- rflutter_alert 2.0.4 [flutter]

transitive dependencies:
- amplify_api_android 0.6.1 [flutter]
- amplify_api_ios 0.6.1 [amplify_core flutter]
- amplify_auth_cognito_android 0.6.1 [flutter]
- amplify_auth_cognito_ios 0.6.1 [amplify_core flutter]
- amplify_core 0.6.1 [aws_common collection flutter intl json_annotation meta plugin_platform_interface uuid]
- amplify_datastore_plugin_interface 0.6.1 [amplify_core collection flutter meta]
- amplify_flutter_android 0.6.1 [flutter]
- amplify_flutter_ios 0.6.1 [amplify_core flutter]
- amplify_storage_s3_android 0.6.1 [flutter]
- amplify_storage_s3_ios 0.6.1 [flutter]
- archive 3.3.0 [crypto path]
- async 2.8.2 [collection meta]
- aws_common 0.1.1 [async collection http meta stream_transform uuid]
- characters 1.2.0
- charcode 1.3.1
- clock 1.1.0
- collection 1.16.0
- cross_file 0.3.3+1 [js meta]
- crypto 3.0.2 [typed_data]
- ffi 2.0.1
- file 6.1.2 [meta path]
- flutter_plugin_android_lifecycle 2.0.6 [flutter]
- flutter_riverpod 1.0.4 [collection flutter meta riverpod state_notifier]
- flutter_web_plugins 0.0.0 [flutter js characters collection material_color_utilities meta vector_math]
- http 0.13.4 [async http_parser meta path]
- http_parser 4.0.1 [collection source_span string_scanner typed_data]
- image_picker_android 0.8.5+1 [flutter flutter_plugin_android_lifecycle image_picker_platform_interface]
- image_picker_for_web 2.1.8 [flutter flutter_web_plugins image_picker_platform_interface]
- image_picker_ios 0.8.5+5 [flutter image_picker_platform_interface]
- image_picker_platform_interface 2.5.0 [cross_file flutter http plugin_platform_interface]
- intl 0.17.0 [clock path]
- js 0.6.4
- json_annotation 4.5.0 [meta]
- material_color_utilities 0.1.4
- meta 1.7.0
- path 1.8.1
- path_provider_android 2.0.16 [flutter path_provider_platform_interface]
- path_provider_ios 2.0.10 [flutter path_provider_platform_interface]
- path_provider_linux 2.1.7 [ffi flutter path path_provider_platform_interface xdg_directories]
- path_provider_macos 2.0.6 [flutter path_provider_platform_interface]
- path_provider_platform_interface 2.0.4 [flutter platform plugin_platform_interface]
- path_provider_windows 2.1.0 [ffi flutter path path_provider_platform_interface win32]
- petitparser 5.0.0 [meta]
- platform 3.1.0
- plugin_platform_interface 2.1.2 [meta]
- process 4.2.4 [file path platform]
- riverpod 1.0.3 [collection meta state_notifier]
- sky_engine 0.0.99
- source_span 1.8.2 [collection path term_glyph]
- state_notifier 0.7.2+1 [meta]
- stream_transform 2.0.0
- string_scanner 1.1.0 [charcode source_span]
- term_glyph 1.2.0
- typed_data 1.3.1 [collection]
- uuid 3.0.6 [crypto]
- vector_math 2.1.2
- win32 2.7.0 [ffi]
- xdg_directories 0.2.0+1 [meta path process]
- xml 6.1.0 [collection meta petitparser]

Device

Android Emulator

OS

Android 8 (development & emulation on Debian Bullseye)

CLI Version

9.1.0

Additional Context

Here's the amplifyconfiguration.dart file that was generated for this project (the only manual changes to this file were adding the blocks tracktunesCognito and lambdaFunctionsCognito at the api section, but nothing else was changed in this file):

const amplifyconfig = ''' {
    "UserAgent": "aws-amplify-cli/2.0",
    "Version": "1.0",
    "api": {
        "plugins": {
            "awsAPIPlugin": {
                "tracktunes": {
                    "endpointType": "GraphQL",
                    "endpoint": "https://??????????????????????????.appsync-api.us-east-1.amazonaws.com/graphql",
                    "region": "us-east-1",
                    "authorizationType": "AWS_IAM"
                },
                "tracktunesCognito": {
                    "endpointType": "GraphQL",
                    "endpoint": "https://??????????????????????????.appsync-api.us-east-1.amazonaws.com/graphql",
                    "region": "us-east-1",
                    "authorizationType": "AMAZON_COGNITO_USER_POOLS"
                },
                "lambdaFunctions": {
                    "endpointType": "REST",
                    "endpoint": "https://??????????.execute-api.us-east-1.amazonaws.com/dev",
                    "region": "us-east-1",
                    "authorizationType": "AWS_IAM"
                },
                "lambdaFunctionsCognito": {
                    "endpointType": "REST",
                    "endpoint": "https://??????????.execute-api.us-east-1.amazonaws.com/dev",
                    "region": "us-east-1",
                    "authorizationType": "AMAZON_COGNITO_USER_POOLS"
                }
            }
        }
    },
    "auth": {
        "plugins": {
            "awsCognitoAuthPlugin": {
                "UserAgent": "aws-amplify-cli/0.1.0",
                "Version": "0.1.0",
                "IdentityManager": {
                    "Default": {}
                },
                "CredentialsProvider": {
                    "CognitoIdentity": {
                        "Default": {
                            "PoolId": "us-east-1:????????-????-????-????-?????????????",
                            "Region": "us-east-1"
                        }
                    }
                },
                "CognitoUserPool": {
                    "Default": {
                        "PoolId": "us-east-1_?????????",
                        "AppClientId": "??????????????????????????",
                        "Region": "us-east-1"
                    }
                },
                "Auth": {
                    "Default": {
                        "authenticationFlowType": "USER_SRP_AUTH",
                        "socialProviders": [],
                        "usernameAttributes": [
                            "EMAIL"
                        ],
                        "signupAttributes": [
                            "EMAIL"
                        ],
                        "passwordProtectionSettings": {
                            "passwordPolicyMinLength": 8,
                            "passwordPolicyCharacters": []
                        },
                        "mfaConfiguration": "OFF",
                        "mfaTypes": [
                            "SMS"
                        ],
                        "verificationMechanisms": [
                            "EMAIL"
                        ]
                    }
                },
                "AppSync": {
                    "Default": {
                        "ApiUrl": "https://??????????????????????????.appsync-api.us-east-1.amazonaws.com/graphql",
                        "Region": "us-east-1",
                        "AuthMode": "AWS_IAM",
                        "ClientDatabasePrefix": "tracktunes_AWS_IAM"
                    },
                    "tracktunes_AMAZON_COGNITO_USER_POOLS": {
                        "ApiUrl": "https://??????????????????????????.appsync-api.us-east-1.amazonaws.com/graphql",
                        "Region": "us-east-1",
                        "AuthMode": "AMAZON_COGNITO_USER_POOLS",
                        "ClientDatabasePrefix": "tracktunes_AMAZON_COGNITO_USER_POOLS"
                    }
                },
                "S3TransferUtility": {
                    "Default": {
                        "Bucket": "??????????????-dev",
                        "Region": "us-east-1"
                    }
                }
            }
        }
    },
    "storage": {
        "plugins": {
            "awsS3StoragePlugin": {
                "bucket": "??????????????-dev",
                "region": "us-east-1",
                "defaultAccessLevel": "guest"
            }
        }
    }
}''';

[HuiSF]

Hello @dorontal Thanks for reporting this issue.

Set the access level to protected for the current user

Quick check: have you signed a user while executing uploading file with protected access level?

[dorontal]

Hi @HuiSF and thank you for the attention to this issue. The answer is yes, a user was signed in at the time of the function call that caused the exception.

My code is set up such that it is only possible to run this upload function when a user is signed in. I know for sure that the user has logged into the app and that Amplify.Auth.getCurrentUser() returns a user right before calling the function that caused the exception.

[HuiSF]

Thanks @dorontal I'll try to repro.

[HuiSF]

I did a quick testing from setting up the S3 category to run the Android App and upload a file. I don't see any issues, the upload request was properly signed.

What does the s3 trigger you mentioned exactly do?

Also have you tested the App in iOS, did it work?

[dorontal]

Hi @HuiSF, and thanks for your attention to this issue. Thanks very much for checking that s3 uploads are okay.

The s3 trigger checks that the file is a valid one and deletes the file if it isn't valid, it also checks that the user has not initiated too many uploads in a time window and doesn't allow the upload to happen if too many uploads have occurred. I see the exact code, this exact trigger, working and running in my Javascript version of the same app.

I have not tested in iOS because I don't have an access to such a device or OS.

I'll post more here if I find out what's causing this - seems like straightforward usage in my code... but will investigate.

[dorontal]

@HuiSF when you tried to reproduce - did you try the CLI setup exactly as I have it above?

The CLI setup is currently the only difference between my code and this documentation - I've even cut & paste their upload code where they generate a dummy file called 'example.txt' and I get the same error thrown as above: access denied.

The S3 trigger is irrelevant here because it does not even get called or invoked yet in any way - the access denied error occurs beforehand.

Since the CLI setup is the only difference between my setup and the documentation, that's why I'm asking if you used the same exact CLI setup when you tried to reproduce things.

[dorontal]

FYI, this issue may be related to #981

[dorontal]

This issue continues more than one month after it has been opened and storage is totally useless for my application still.

Just tried it with the new version of the CLI, version 9.2.1 - same error reported

[log] Error uploading file: StorageException(message: Something went wrong with your AWS S3 Storage upload file operation, recoverySuggestion: See attached exception for more information and suggestions, underlyingException: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: N8J1A3Y77KY9X89D), S3 Extended Request ID: BZO944KC26BtE92xZ876V3NcuyHv6MOAnKw1j9aTpaYIO54drcOO7RY560YhFcam+3NSfGO7AXA=)

I would appreciate confirmation if anybody else sees access denied exceptions on uploads when using storage after setting it up via the CLI for

• Guests - read access
• Authorized users - create, update, read, write access

@Jordan-Nelson or @HuiSF -- when you tried to replicate, did you replicate my CLI setup as I reported above? I.e, did you give these permissions?

• Guests - read access
• Authorized users - create, update, read, write access

in your replication attempt?

This would have been necessary to properly try to replicate - at least follow the same CLI setup that I have, and I am just trying to make sure that the replication attempt - and the subsequent abandonment of this issue, which happened very quickly - are for good reasons.

[Jordan-Nelson]

Hi @dorontal - I apologize that there was no update on this issue for a while. I have just attempted to reproduce this using the steps you provided to set up storage via the CLI. I see files upload correctly. I am attempting to reproduce this with the code you provided, and I am using an Android emulator with the same API level that you are. Unfortunately I am unable to reproduce this.

I see that you added a lambda trigger when setting up storage via the CLI, but did not edit it. Have you edited this lambda at all since?

Do you have access to an iPhone or iPhone emulator? Seeing if it works on an iPhone issue might help narrow down the issue.

[dorontal]

Hi @Jordan-Nelson I have not edited the lambda and the exception seems to happen before the lambda even gets invoked as this is an access permissions error. The dashboard of AWS lambda does not show any invocation attempts after this exception is thrown.

When you tried to replicate - did you follow the same permissions for Authorized users and Guests that I used in my setup above? In the setup above, authorized users get CRUD access while guests get only read access. The exception shows up when an authorized user is signed in and tries to upload something to any of "private", "protected" or "public" path buckets.

[Jordan-Nelson]

@dorontal - Yes, I used those same permissions.

Would you be able to confirm the user that us currently logged in is able to call auth APIs such as Amplify.Auth.getCurrentUser()?

Are you working in an environment where you would be able to remove/delete that storage bucket (amplify remove storage) and re-add it?

[dorontal]

Thanks, @Jordan-Nelson, for confirming that you used the same setup! I've just inserted a call to Auth.getCurrentUser() right before the call to Storage.uploadFile() and it returns the current user without issue, right before the call to storage that throws the exception.

[ Just noticed that the onProgress callback that I used just to log the progress logs Fraction completed: 1.0, which is interesting. ]

I then removed the storage module altogether as you suggest with amplify remove storage and re-added it (using an expect script to add that module automatically with all the CLI questions and answers, exactly the same way every time) and I see the exact same exception thrown:

[log] Fraction completed: 1.0
[log] Error uploading file: StorageException(message: Something went wrong with your AWS S3 Storage upload file operation, recoverySuggestion: See attached exception for more information and suggestions, underlyingException: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: QW8GD8WNQNHM5VY5), S3 Extended Request ID: FPMMYiyhJecR+bV1bTpq4nj34Y5HfMFzzo2l1UAHNyD9yqUxUdbUWdKBpI3B5YYoPR7ogN80xuE=)

Also, I've noticed that the remaking of the storage module created a new storage trigger function that is blank - of course just as before, this blank function never gets invoked - not in the S3 dashboard, not in cloudwatch logs. I am pretty sure that the S3 trigger has nothing to do with this issue because as you can see the same exception persists with a 100% generic code-generated trigger.

Also, I've noticed that in amplifyconfiguration.dart the Storage configuration has

                "defaultAccessLevel": "guest"

Is it possible that the Amplify libraries think that a guest is trying the call to Storage.uploadFile() even though a user is currently signed in? How does the library code determine if it's a guest or a signed in user trying to make the uploadFile() call?

[dorontal]

@HuiSF and @Jordan-Nelson -- issue solved! Thank you so much for your attention to this issue and please accept my apologies for thinking it was something with either of your reproductions of the issue, it wasn't.

The line that gave me the clue was printed by the Amplify CLI at the end of amplify update storage - that CLI command said:

...
✅ Successfully added resource s3 locally

⚠️ If a user is part of a user pool group, run "amplify update storage" to enable IAM group policies for CRUD operations
✅ Some next steps:
...

I had never run the suggested amplify update storage before as it suggests to do at the end of amplify add storage as shown above. In my case, every authenticated user that gets added is automatically given a group - the everyone group (there are reasons for this that have to do with fine level auth control needed elsewhere and that's a different discussion).

Automatically adding authenticated users to a group is what caused the issue.

When your authenticated users get added to a group automatically, you must run an additional amplify update storage and explicitly give users of the added group the permissions needed, as done in the following CLI transcript, which is the command that finally caused uploads to succeed and this issue to disappear:

dtal@thing4: amplify update storage
? Select from one of the below mentioned services: Content (Images, audio, video, etc.)
✔ Restrict access by? · Both
✔ Who should have access: · Auth and guest users
✔ What kind of access do you want for Authenticated users? · create/update, read, delete
✔ What kind of access do you want for Guest users? · read
✔ Select groups: · everyone
✔ What kind of access do you want for everyone users? · create/update, read, delete
✔ Select from the following options · Update the Trigger
✔ Select from the following options · Create a new function
✅ Successfully added resource S3Triggerb0e8dc02 locally
✔ Do you want to edit the local S3Triggerb0e8dc02 lambda function now? (y/N) · no
✅ Successfully updated resource

I no longer get the 403 unauthorized error.

[Jordan-Nelson]

@dorontal - I am glad you were able to resolve the issue. Thanks for the detailed info that you provided on what caused this and how to resolve it. It will likely help someone else in the future that runs into this same issue.

I am going to close this issue since you have resolved it.

[devengineergit]

@HuiSF and @Jordan-Nelson -- issue solved! Thank you so much for your attention to this issue and please accept my apologies for thinking it was something with either of your reproductions of the issue, it wasn't.

The line that gave me the clue was printed by the Amplify CLI at the end of amplify update storage - that CLI command said:

...
✅ Successfully added resource s3 locally

⚠️ If a user is part of a user pool group, run "amplify update storage" to enable IAM group policies for CRUD operations
✅ Some next steps:
...

I had never run the suggested amplify update storage before as it suggests to do at the end of amplify add storage as shown above. In my case, every authenticated user that gets added is automatically given a group - the everyone group (there are reasons for this that have to do with fine level auth control needed elsewhere and that's a different discussion).

Automatically adding authenticated users to a group is what caused the issue.

When your authenticated users get added to a group automatically, you must run an additional amplify update storage and explicitly give users of the added group the permissions needed, as done in the following CLI transcript, which is the command that finally caused uploads to succeed and this issue to disappear:

dtal@thing4: amplify update storage
? Select from one of the below mentioned services: Content (Images, audio, video, etc.)
✔ Restrict access by? · Both
✔ Who should have access: · Auth and guest users
✔ What kind of access do you want for Authenticated users? · create/update, read, delete
✔ What kind of access do you want for Guest users? · read
✔ Select groups: · everyone
✔ What kind of access do you want for everyone users? · create/update, read, delete
✔ Select from the following options · Update the Trigger
✔ Select from the following options · Create a new function
✅ Successfully added resource S3Triggerb0e8dc02 locally
✔ Do you want to edit the local S3Triggerb0e8dc02 lambda function now? (y/N) · no
✅ Successfully updated resource

I no longer get the 403 unauthorized error.

You are a lifesaver. Struggled with this issue for hours. None in the documentation. Thanks much!

[abdallahshaban557]

@devengineergit - thank you for the message on this issue. We will update our documentation to reduce confusion for future developers!