[storage] Remove file is not working

[iosephmagno]

Description

After updating to amplify_storage_s3: 2.4.1, Amplify.Storage.remove() is no longer deleting files from AWS s3.

Categories

• [ ] Analytics
• [ ] API (REST)
• [ ] API (GraphQL)
• [ ] Auth
• [ ] Authenticator
• [ ] DataStore
• [ ] Notifications (Push)
• [X] Storage

Steps to Reproduce

Here is the code

try {
      final result = await Amplify.Storage.remove(path: StoragePath.fromString('public/$_key')).result;
      debugPrint(
        'Deleted file from AWS S3: ${result.removedItem.path}',
      );
      return true;
    } on StorageException catch (e) {
      debugPrint('Error deleting file from AWS S3: $e');
    }

Method run successfully, no error.

Logs print: I/flutter (20655): Deleted file from AWS S3: public /avatars/3a497312-781f-4678-8bf1-7df0826c51df1701729021307905566_profile_image.jpg

But the file public /avatars/3a497312-781f-4678-8bf1-7df0826c51df1701729021307905566_profile_image.jpg is still listed inside AWS s3.

Screenshots

No response

Platforms

• [X] iOS
• [X] Android
• [ ] Web
• [ ] macOS
• [ ] Windows
• [ ] Linux

Flutter Version

3.27.0-1.0.pre.31 • channel master

Amplify Flutter Version

2.4.1

Deployment Method

Amplify Gen 2

Schema

No response

[NikaHsn]

@iosephmagno sorry that you are facing this issue. I was not able to reproduce this issue.

re: But the file public /avatars/3a497312-781f-4678-8bf1-7df0826c51df1701729021307905566_profile_image.jpg is still listed inside AWS s3.

How do you confirm whether the file has been deleted from the S3 bucket?

[iosephmagno]

Hello @NikaHsn sorry for the belated reply, but the notification email got lost in my inbox. I simply check inside s3 folder (refresh folder page) from console and the file is still listed there after it is removed by the app.

Any idea about what the problem might be?

[khatruong2009]

Hi @iosephmagno, does the name of the bucket that you're checking in the console also match the bucket name that's in your amplify_outputs.dart file?

[iosephmagno]

Yes we upload files successfully, we just cannot delete them.

[iosephmagno]

But the delete code shared anove returns success (no StorageException). Can you please check this thoroughly? Not being able to delete users' files is a severe bug for us, apart obvious consideration related to the increasing storage size, there are Laws to respect (if user deletes a file, the file must be deleted on our backend).

[khatruong2009]

This probably shouldn't have an effect on this API but could you try it on the flutter stable channel as well?

[iosephmagno]

@khatruong2009 it is the same. Can you guys test again the new api?

[iosephmagno]

You introduced path which I think might be involved with the issue

[Equartey]

Hi @iosephmagno, we could not reproduce the observed behavior.

I tried with the following snippet and confirm the file was removed from the console:

await Amplify.Storage.remove(
  path: StoragePath.fromString(path),
).result;

Can you please provide your storage backend configuration?

[Equartey]

@iosephmagno also which amplify_storage_s3 version did you upgrade from?

[iosephmagno]

Hello, it is version 2.4.1

[khatruong2009]

@iosephmagno which version were you using before?

[iosephmagno]

Hello, previous version was 1.7.0

To reproduce the issue:

1. create a subfolder "test" inside "public"
2. upload 2 files "file1.jpg" and "file2.jpg" to "test"
3. delete "file1.jpg"
4. browse folder "public/test" from web console
5. In our case, "file1.jpg" is still there.

To delete the file, we use the code attached above.

[Equartey]

@iosephmagno, Can you please provide your gen 2 amplify storage backend definition?

This is typically located in [project_root]/amplify/storage/resource.ts

[iosephmagno]

hello @Equartey , the only .ts file I see is inside amplify/backend/types/amplify-dependent-resources-ref.d.ts and it is:

export type AmplifyDependentResourcesAttributes = {}

Other files inside amplify folder are:

amplify/team-provider-info.json is

{
  "dev": {
    "awscloudformation": {
      "AuthRoleName": "our-value-here",
      "UnauthRoleArn": "arn:aws:iam::793017513442:role/amplify-our-value-here",
      "AuthRoleArn": "arn:aws:iam::793017513442:role/amplify-our-value-here",
      "Region": "us-east-1",
      "DeploymentBucketName": "amplify-our-value-here",
      "UnauthRoleName": "amplify-our-value-here",
      "StackName": "amplify-our-value-here",
      "StackId": "arn:aws:cloudformation:us-east-1:793017513442:stack/our-value-here",
      "AmplifyAppId": "our-value-here"
    },
    "categories": {
      "auth": {
        "presence8640700c": {
          "userPoolId": "our-value-here",
          "userPoolName": "our-value-here",
          "webClientId": "our-value-here",
          "nativeClientId": "our-value-here",
          "identityPoolId": "us-east-1:our-value-here",
          "identityPoolName": "our-value-here",
          "allowUnauthenticatedIdentities": true,
          "authRoleArn": "our-value-here",
          "authRoleName": "our-value-here",
          "unauthRoleArn": "our-value-here",
          "unauthRoleName": "our-value-here"
        }
      },
      "storage": {
        "our-value-here": {
          "bucketName": "our-value-here",
          "region": "us-east-1"
        }
      }
    }
  }
}

amplify/backend/backend-config.json is

{
  "auth": {
    "our-value-here": {
      "service": "Cognito",
      "serviceType": "imported",
      "providerPlugin": "awscloudformation",
      "dependsOn": [],
      "customAuth": false
    }
  },
  "storage": {
    "our-value-here": {
      "service": "S3",
      "serviceType": "imported",
      "providerPlugin": "awscloudformation",
      "dependsOn": []
    }
  }
}

amplify/backend/storage/our-value-here/parameters.json is

{
  "resourceName": "our-value-here",
  "serviceType": "imported"
}

amplify/backend/auth/our-value-here/parameters.json is

{
  "authSelections": "identityPoolAndUserPool",
  "resourceName": "our-value-here",
  "serviceType": "imported",
  "region": "us-east-1",
  "requiredAttributes": [
    "email"
  ],
  "passwordPolicyMinLength": 8,
  "passwordPolicyCharacters": [],
  "mfaConfiguration": "OFF",
  "autoVerifiedAttributes": [
    "email"
  ],
  "mfaTypes": []
}

amplify/.config/project-config.json is

{
  "projectName": "our-value-here",
  "version": "3.1",
  "frontend": "flutter",
  "flutter": {
    "config": {
      "ResDir": "./lib/"
    }
  },
  "providers": [
    "awscloudformation"
  ]
}

amplify/backend/tags.json is

[
  {
    "Key": "user:Stack",
    "Value": "{project-env}"
  },
  {
    "Key": "user:Application",
    "Value": "{project-name}"
  }
]

amplify/cli.json is

{
  "features": {
    "graphqltransformer": {
      "addmissingownerfields": true,
      "improvepluralization": false,
      "validatetypenamereservedwords": true,
      "useexperimentalpipelinedtransformer": true,
      "enableiterativegsiupdates": true,
      "secondarykeyasgsi": true,
      "skipoverridemutationinputtypes": true,
      "transformerversion": 2,
      "suppressschemamigrationprompt": true,
      "securityenhancementnotification": false,
      "showfieldauthnotification": false
    },
    "frontend-ios": {
      "enablexcodeintegration": true
    },
    "auth": {
      "enablecaseinsensitivity": true,
      "useinclusiveterminology": true,
      "breakcirculardependency": true,
      "forcealiasattributes": false,
      "useenabledmfas": true
    },
    "codegen": {
      "useappsyncmodelgenplugin": true,
      "usedocsgeneratorplugin": true,
      "usetypesgeneratorplugin": true,
      "cleangeneratedmodelsdirectory": true,
      "retaincasestyle": true,
      "addtimestampfields": true,
      "handlelistnullabilitytransparently": true,
      "emitauthprovider": true,
      "generateindexrules": true,
      "enabledartnullsafety": true
    },
    "appsync": {
      "generategraphqlpermissions": true
    },
    "latestregionsupport": {
      "pinpoint": 1,
      "translate": 1,
      "transcribe": 1,
      "rekognition": 1,
      "textract": 1,
      "comprehend": 1
    },
    "project": {
      "overrides": true
    }
  }
}

[iosephmagno]

As mentioned issue is with deleting files only.

[iosephmagno]

@Equartey this is our S3 bucket policy, which allow both upload and delete to cognito user amplify-our-user-unauthRole. Since upload works, delete should work as well.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ListObjectsInBucket",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::our-bucket”
        },
        {
            "Sid": "AllObjectActions",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::our-bucket/*"
        },
        {
            "Sid": "Allow upload to cognito users",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::793017513442:role/amplify-our-user-unauthRole"
            },
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::our-bucket/*"
        },
        {
            "Sid": "Allow delete only to cognito users",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::793017513442:role/amplify-our-user-unauthRole"
            },
            "Action": [
                "s3:DeleteObject",
                "s3:DeleteObjectVersion"
            ],
            "Resource": "arn:aws:s3:::our-bucket/*"
        }
    ]
}

[ekjotmultani]

Hi @iosephmagno, thank you for providing your backend definition and bucket policy. We will investigate this further with the given information.

[iosephmagno]

@Equartey thxs, if possible, please prioritize this issue because deleting UGC files is a law obligation for us.

Also, the below code should throw an error if file is not deleted successfully, but we get no error. So, if we didn't check the AWS bucket manually, we would have never discovered the issue.

try {
      final result = await Amplify.Storage.remove(path: StoragePath.fromString('public/$_key')).result;
      debugPrint(
        'Deleted file from AWS S3: ${result.removedItem.path}',
      );
      return true;
    } on StorageException catch (e) {
      debugPrint('Error deleting file from AWS S3: $e');
    }

[ekjotmultani]

Hello @iosephmagno, for clarification, how are you setting up your s3 storage backend? Are you using Amplify CLI Gen 1 or existing resources? When we attempt to use the supplied policy, we get the following errors:

Unsupported Principle: the policy type IDENTITY_POLICY does not support the Principle element. Remove the Principle element.

Unsupported SID: Update the characters in the SID element to use one of the following character types: [a-z, A-Z, 0-9].

[iosephmagno]

Hello @Equartey, we followed the package setup instruction back then and after that moment on we simply updated the amplify plugin inside pubspec. If any change or checks must be done, can you please provide precise instructions? The info you wrote in the comment above about "gen1 vs existing resources" and the error about "Unsupported Principle / SID" are criptic to us.

[iosephmagno]

As for the s3 policy, we get no error with above bucket policy. We have an amplify-our-user-unauthRole set in our IAM and then we use it to grant file upload/deletion rights to only this user. It works for object upload, it fails for object deletion (with no error thrown on the client by the plugin).

[iosephmagno]

@Equartey hello, we've made more tests and discovered that we can delete the file. The issue was due to a wrong path, and not to AWS settings.

However, there is still an issue to solve, because Amplify.Storage.remove doesn't throw error if path is wrong (no object found for _key), so the developer cannot understand that the deletion failed and why it failed. try { final result = await Amplify.Storage.remove(path: StoragePath.fromString('$_key')).result; } on StorageException catch (e) { debugPrint('$_debugPrefix Error deleting file from AWS S3: $e'); }

This is the reason why we thought that there was an issue with the plugin, we just couldn't understand that _key was wrong.

[ekjotmultani]

Hi @iosephmagno it's great to hear that the delete is working! It is the expected behaviour that no error is thrown from s3 when attempting to remove an object that doesn't exist.

[iosephmagno]

Hello @ekjotmultani, could you guys throw an error for this? Without it, we won’t be able to debug effectively if an issue arises. File deletion is essential for legal compliance, so it’s important to know if a file fails to delete for any reason. Thx!

[tyllark]

Helllo @iosephmagno, I can mark this issue as a feature request, but currently S3 responses for DeleteObject don't distinguish between a existing and non-existing path (unless there are permission or an invalid path format).