[v6] How to set dynamic custom prefix for each file upload like v5

[natuan62]

Before opening, please confirm:

• [X] I have searched for duplicate or closed issues and discussions.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

Vue

Amplify APIs

Storage

Amplify Version

v6

Amplify Categories

storage

Backend

None

Environment information

# Put output below this line
  System:
    OS: macOS 14.4.1
    CPU: (8) arm64 Apple M1
    Memory: 103.72 MB / 16.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 20.11.1 - ~/.nvm/versions/node/v20.11.1/bin/node
    Yarn: 1.22.19 - ~/.yarn/bin/yarn
    npm: 7.20.0 - ~/.config/yarn/global/node_modules/.bin/npm
    pnpm: 8.15.7 - ~/Library/pnpm/pnpm
  Browsers:
    Chrome: 124.0.6367.61
    Safari: 17.4.1
  npmPackages:
    fast-xml-parser: ^4.3.2 => 4.3.2
    mime-types: ^2.1.35 => 2.1.35
    sharp: ^0.32.6 => 0.32.6
    svg-parser: ^2.0.4 => 2.0.4
    typescript: ^5.3.2 => 5.3.2
    unzipper: ^0.10.14 => 0.10.14
  npmGlobalPackages:
    @angular/cli: 10.1.2
    @aws-amplify/cli: 12.10.1
    corepack: 0.23.0
    npm: 10.2.4

Describe the bug

When upload multi file and I want to config prefix for each file like v5 but it seem not working

Expected behavior

prefixResolver working when pass dynamic prefix

Reproduction steps

1. Select upload file with multi files
2. Set dynamic prefixResolver for each file
3. File upload to correct prefix

• and in v5 I can use configure like this

Storage.configure({
          customPrefix: {
            public: 'public/',
            protected: 'protected/',
            private: getPrefixByType(file.type),
          },
        });

await Storage.put(....)

Code Snippet

// Put your code below this line.

import { uploadData, downloadData, TransferProgressEvent } from 'aws-amplify/storage';

  const uploadS3 = async (input: {file: string, key: string}, prefix: string) => {
    const libraryOptions: any = {
      Storage: {
        S3: {
          prefixResolver: async (input: { accessLevel: AccessLevel, targetIdentityId: string}) => {
            const { targetIdentityId } = input;
            if (input.accessLevel === 'guest') {
              return 'publicPrefix/';
            } else if (input.accessLevel === 'protected') {
              return `protected/${targetIdentityId}/`;
            }
            // although prefix input is dynamic but prefix always not change, still is same value
            return `private/${prefix}/${targetIdentityId}/`;
          },
        },
      },
    };
    Amplify.configure(awsconfig as ResourcesConfig, libraryOptions);

    const operation = uploadData({
      key: input.key,
      data: input.file,
      options: {
        contentType: 'text/plain',
        accessLevel: 'private',
      },
    });
    const result = await operation.result;
  };

  const listFiles = [{file: 'video1', type: 'video'}, {file: 'image1', type: 'image'}];

  await Promise.all(listFiles.map((file, index) => {
    const prefix = file.type === 'video' ? 'video' : 'image';
    return uploadS3({file: file.file, key: file.file}, prefix);
  }

Log output

// Put your logs below this line

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

[natuan62]

I found this solution, seem it not good. Anyway better this way @cwomack ?

const uploadS3 = async (input: {file: string, key: string}, accessLevel: AccessLevel) => {
    const libraryOptions: any = {
      Storage: {
        S3: {
          prefixResolver: async (input: { accessLevel: AccessLevel, targetIdentityId: string}) => {
            const { targetIdentityId } = input;
           
            const prefix = input.accessLevel === 'private' ? 'video' : 'image';
            return `private/${prefix}/${targetIdentityId}/`;
          },
        },
      },
    };
    Amplify.configure(awsconfig as ResourcesConfig, libraryOptions);

    const operation = uploadData({
      key: input.key,
      data: input.file,
      options: {
        contentType: 'text/plain',
        accessLevel,
      },
    });
    const result = await operation.result;
  };

  const listFiles = [{file: 'video1', type: 'video'}, {file: 'image1', type: 'image'}];

  await Promise.all(listFiles.map((file, index) => {
    const accessLevel = file.type === 'video' ? 'private' : 'protected';
    return uploadS3({file: file.file, key: file.file}, accessLevel);
  }

[cwomack]

Hey, @natuan62 👋. We appreciate you opening this issue and I'm going to mark it as a feature request for v6 of Amplify for version parity from v5. Previously, you could use something like the file type to create those dynamic prefixes in S3 upon uploading.

For v6 (and using Gen 1 with CLI created backend resources), it looks like we may need to do something along the lines of exposing extra arguments for a custom prefixResolver(), such as file type, and make this optional. However, in the Gen 2 Developer Experience (which is currently in developer preview) this should be supported once we go General Availability. The functionality isn't released yet, but we will follow up on this issue as soon as it is.

As for the code snippet for a a solution you provided above, there's a risk of causing concurrent calls to S3 calls in your promise callback, which could lead to potential race conditions. If the uploaded file is larger than 5 MB, it will call the multi-part upload and make multiple calls within the promise. I'm concerned that you may end up with different calls leading to different prefixes if this happens.

Best recommendation here would be to await the release of this functionality in the Gen2 Developer Experience, or until the feature request is fully supported on the Gen 1 side of things.