Add baseUrl and rootPath properties to upload-aws-s3 plugin's configuration

[v-dimitroff]

What does it do?

This PR allows the upload-aws-s3 plugin to be configured to work with a bucket served through a CDN. It provides the interface to specify the domain of the CDN through the baseUrl variable and also pick a specific path inside the CDN to store assets from Strapi with the rootPath variable

Why is it needed?

It is a very common case users will already have some content and assets served from a CDN and integration with Strapi should not break this set up.

How to test it?

Related issue(s)/PR(s)

Opened as a replacement to the #11886 issue due to inactive contributor.

[strapi-cla]

All committers have signed the CLA.

[Warxcell]

I would suggest decoupling the actual upload and url generation and even filename generation.

[alexandrebodin]

@v-dimitroff Let me know if you need some guidance to add some testing to this PR :)

[derrickmehaffy]

If we are doing this for AWS, should we also do it for cloudinary or other providers?

[WardenCommander]

Any chance for this to be in next strapi version?

[Warxcell]

If we are doing this for AWS, should we also do it for cloudinary or other providers?

That's why I suggest to decouple url and filename generation and the upload itself. That way - you do it once, and no matter where you upload files - it works. Same for the name - you can use custom naming of files, without touching the upload itself.

Basically upload config could looks something like

{
   namingStrategy: (entity) => someFunctionThatGeneratesFilenameOfUpload,
   provider: (entity, path) => someFunctionThatUploads <= second argument is result of namingStrategy fn
   urlResolver: (entity, path) => someFunctionThatGeneratesUrlOfUpload  <= second argument is result of namingStrategy fn
}

And you can replace any part of upload process without changing other.

Full example:

import { AwsS3Provider } from "@strapi/aws-s3-provider"

export default {
   namingStrategy: (entity) => `my-prefix/${entity.hash}${entity.ext}`
   provider: new AwsS3Provider({ accessKeyId, secretAccessKey, region, bucket} ),
   urlResolver: (entity, path) => `http://mycustom-url/some-prefix/${path}`
}

[dzmitry-kankalovich]

I couldn't wait till this is merged, so for those who need this ASAP there is a way to get around, basically create your custom provider and place it as NPM package under @strapi in your node_modules.

You can follow example here and use the source code from this PR. One thing though - for v4 it wouldn't work as-is, I needed to additionally specify local dependency in package.json:

// ...
  "dependencies": {
     // ...
    "@strapi/strapi": "4.1.9",
    "@strapi/provider-upload-aws-s3-with-cdn": "file:./providers/strapi-provider-upload-aws-s3-with-cdn",
    // ...
  },
// ...

Then npm install would symlink your local package under the node_modules and strapi would be able to find and use the provider.

[lissau]

@petersg83 Could you review this please or assign someone else?

[v-dimitroff]

@petersg83 Could you review this please or assign someone else?

Hey guys, I'm glad there is some recent activity here - more than happy to finish the work once we have clear requirements of the solution because the scope has changed slightly than initially planned.

[diestrin]

I'm also very interested in this feature. Is the planning looking far in the schedule?

[ihmpavel]

Hi,

In January I created custom local provider based on S3, which is handling Cloudfront.

I ended invalidating cache, because replacing the image do not purge the Cloudfront cache (the name of the file does not change) and users will see the old version until the cache invalidation (few days/hours/...).

I created something like

init(config) {
  const CloudFront = new AWS.CloudFront({
    apiVersion: '2020-05-31',
    ...config,
  })
  ...

delete(file) {
  // delete image
  ...
  // if there is CF attached, create invalidation
  await CloudFront.createInvalidation({
    DistributionId: config.cloudfrontDistributionId,
    InvalidationBatch: {
      CallerReference: Date.now().toString(),
      Paths: {
        Quantity: 1,
        // get just the file key, without cdn in url
        Items: [file.url.replace(config.cdn, '')],
      },
    },
  }).promise();
  ...
}

I suggest adding this to prevent issues, why is not image changing while using Cloudfront

Edit: only applicable to Cloudfront

[alexandrebodin]

Hey @v-dimitroff do you think you will be able to finish this at some point?

[v-dimitroff]

Hey @v-dimitroff do you think you will be able to finish this at some point?

Yeah, sorry for the delay, things didn't come my way the past weeks. I'll be right on it tomorrow and make sure to finish the job!

[JWesorick]

Just want to throw my support in behind this PR. I can really use this.

[v-dimitroff]

@Sija @JWesorick @alexandrebodin Guys, if you have the time - please go forward with this one. Really wanted to to finish the work but it seems I just can't get back to the project. So if you have the time just go on and sorry for being a bottle neck so long.

[kristianfreeman]

Also want to chime in here - this PR landing would allow me to build some solutions to the problem of Strapi only allowing public bucket/asset access, so having this land would be great. Any update from last month?

[dobrud]

I'm also depending on this feature. I'm very happy to spare some time finishing this feature. @v-dimitroff May I ask you to give me access to your repository to work on this branch/PR?

EDIT: In the meantime I just opened https://github.com/strapi/strapi/pull/14722. It's based on this branch, addresses all open MR and adds some tests

[v-dimitroff]

I'm also depending on this feature. I'm very happy to spare some time finishing this feature. @v-dimitroff May I ask you to give me access to your repository to work on this branch/PR?

EDIT: In the meantime I just opened #14722. It's based on this branch, addresses all open MR and adds some tests

Hey, thanks for tuning in! Not sure if my branch is useful but invited you as a collaborator!

[dobrud]

@v-dimitroff Thanks for granting access! To keep the change history, I pushed all my changes to this branch and @alexandrebodin closed https://github.com/strapi/strapi/pull/14722 👍

[codecov[bot]]

Codecov Report

Patch coverage: 81.81% and no project coverage change.

Comparison is base (7536fb5) 59.95% compared to head (5a0caeb) 59.95%.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #13040   +/-   ##
=======================================
  Coverage   59.95%   59.95%           
=======================================
  Files        1484     1484           
  Lines       37346    37353    +7     
  Branches     7480     7484    +4     
=======================================
+ Hits        22389    22394    +5     
- Misses      12769    12771    +2     
  Partials     2188     2188           

Flag	Coverage Δ
back	50.07% <81.81%> (+0.01%)	:arrow_up:
front	66.21% <ø> (ø)
unit_back	50.07% <81.81%> (+0.01%)	:arrow_up:
unit_front	66.21% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
packages/providers/upload-aws-s3/lib/index.js	71.79% <81.81%> (+5.12%)	:arrow_up:

... and 2 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[limichange]

This configuration feature actually passed a full year.

[nicoes]

@alexandrebodin are you still reviewing changes or can this be merged? :)

[alexandrebodin]

Hi 👋 back from a long leave I'll review this again !

[dominikzogg]

@v-dimitroff thanks, using the rootPath variable and could replace my own custom provider thanks to this