gatsby-plugin-s3

gatsby-plugin-s3 copied to clipboard

Please support specifying a prefix path in the destination bucket. When using CloudFront, it is possible to serve content from a subpath in a bucket, and there are use cases that require this with direct S3 websites as well.

The same bucket might house multiple variants of the same website, or access logs, or anything else in addition to the website.

Good idea. Gonna look into this.

This would be great as well so that larger uploads wouldn't be officially rolled out until they are entirely complete. My current build has a static folder with about 4gb worth of assets so as the deploy goes out there is a good amount of time where it is not complete and the index files are referencing broken assets.

With the deploy having the ability to specify a prefix path, the best use case for me would be to use it as versioning.

@drewbietron Using this for versioning sounds interesting, but my understanding is that if a static file (e.g. an image) hasn't changed then it will have the same file name. gatsby-plugin-s3 should detect that the file already exists (and has the same checksum) and skip uploading that file. Is it uploading 4GB every time you deploy? I don't think that should happen.

Is there any progress on getting that branch merged? I'm in need of this change 😄

Right now in it's current state it will overwrite the existing routing rules, so if you have multiple gatsby sites in the same bucket, it will keep overriding the most previous deployed gatsby site's redirects, so I still need to make it merge with any existing rules.
Additionally, the tests appear to be failing, haven't really looked into as of why yet.

You're welcome to fork the feature/prefix branch and continue work on it, I'm likely pretty busy this week sadly.

I've been running the branch for the past week and it seems to be working decently. I only have a single site, but I can confirm the main features of it work. I don't really have time to look into the merging of multiple sites sorry.

If the branch seems to be working fine, when do you guys think it could get raised as a PR and be merged?

Released in 0.3.0

Hi, thank you for this plugin, I'm not able to make this feature work here is my config

{
  pathPrefix: `/v2`,
  siteMetadata: {
    title: ``,
    description: `We Got Your App`,
    author: `@`,
    siteUrl: ``,
  },
  plugins: [
    `gatsby-plugin-react-helmet`,
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `images`,
        path: `${__dirname}/src/images`,
      },
    },
    `gatsby-transformer-sharp`,
    `gatsby-plugin-sharp`,
    `gatsby-plugin-layout`,
    `gatsby-plugin-sass`,
    `gatsby-plugin-antd`,
    `gatsby-plugin-sitemap`,
    `gatsby-plugin-robots-txt`,
    {
      resolve: `gatsby-plugin-manifest`,
      options: {
        name: `site-new`,
        short_name: ``,
        start_url: `/v2`,
        background_color: `#663399`,
        theme_color: `#663399`,
        display: `minimal-ui`,
        icon: `src/images/ms-icon-310x310.png`, // This path is relative to the root of the site.
      },
    },
    {
      resolve: `gatsby-plugin-s3`,
      options: {
        params: {
          "fonts/**/**.ttf": {
            CacheControl: "public, max-age=31536000, immutable",
          },
          "**/**.ttf": {
            CacheControl: "public, max-age=31536000, immutable",
          },
          "**/**/**.ttf": {
            CacheControl: "public, max-age=31536000, immutable",
          },
        },
        bucketName: process.env.S3_BUCKET_NAME,
        pathPrefix: `/v2`,
        region: process.env.S3_BUCKET_REGION,
        acl: null,
      },
    },
    {
      resolve: "gatsby-plugin-web-font-loader",
      options: {
        custom: {
          families: ["Ubuntu", "alwyn-new-web"],
          urls: ["/fonts/ubuntu/ubuntu.css", "/fonts/alwyn-new/alwyn.css"],
        },
      },
    },
    {
      // ONLY WORKS AFTER THE SITE IS BUILT FOR PROD/STAGING NOT IN DEVELOPMENT MODE
      // TO TEST DO gatsby build && gatsby serve
      resolve: "gatsby-plugin-hubspot",
      options: {
        trackingCode: process.env.HUBSPOT_KEY,
      },
    },
    {
      resolve: `gatsby-source-contentful`,
      options: {
        spaceId: process.env.CONTENTFUL_SPACE_ID,
        accessToken: process.env.CONTENTFUL_ACCESS_TOKEN,
        downloadLocal: true,
      },
    },
    // this (optional) plugin enables Progressive Web App + Offline functionality
    // To learn more, visit: https://gatsby.dev/offline
    {
      resolve: `gatsby-plugin-offline`,
      options: {
        runtimeCaching: [
          {
            // Use cacheFirst since these don't need to be revalidated (same RegExp
            // and same reason as above)
            urlPattern: /(\.js$|\.css$|static\/)/,
            handler: `cacheFirst`,
          },
          {
            // Add runtime caching of various other page resources
            urlPattern: /^https?:.*\.(png|jpg|jpeg|webp|svg|gif|ttf|tiff|js|woff|woff2|json|css)$/,
            handler: `staleWhileRevalidate`,
          },
          {
            // Google Fonts CSS (doesn't end in .css so we need to specify it)
            urlPattern: /^https?:\/\/fonts\.googleapis\.com\/css/,
            handler: `staleWhileRevalidate`,
          },
        ],
      },
    },
  ],
}

Is there something wrong in there? i always get 404 in all assets when trying to hit the bucket

@jariz it looks like what changed in 0.3.0 was supporting the gatsby pathPrefix variable. While this is a useful feature, I think this is different that the original ask here which I believe was more about a deploy prefix.

Example use case:

• CloudFront staging.example.com uses my-deploys bucket with /staging as the origin path.
• CloudFront featureA.example.com uses my-deploys bucket with /featureA or [sha1] as the origin path.

Or.. the initial use case @nakedible had:

• CloudFront www.example.com uses website-bucket bucket with /static as the origin path.
• Access logs are also stored in website-bucket under /logs/
• Uploads are also stored in website-bucket under /tmp-data/
• Etc..

These are all independent from the gatsby prefixPath.

Hi,

I've set the Gatsby pathPrefix, but it's not taken into account when deploying, ie my content is deployed at the root, not in my prefix's location. I'm using version 0.3.1.

Note: the prefix is working fine with gatsby serve, so I figured the problem might come from this plugin.

Regards

Hi,

I've set the Gatsby pathPrefix, but it's not taken into account when deploying, ie my content is deployed at the root, not in my prefix's location. I'm using version 0.3.1.

Note: the prefix is working fine with gatsby serve, so I figured the problem might come from this plugin.

Regards

Echoing my man's sentiment right here. I'm having this same exact issue. I'm using 0.3.2 Some help would be great.

Let's reopen this because of all the reports. I'll take a closer look into this soon.

I found a workaround for this issue that doesn't involve changing the plugin.

Taking, for instance, the destination /v2

{
  "deploy": "mv public public-temp && mkdir -p public && mv public-temp public/v2 && gatsby-plugin-s3 deploy --yes"
}

The workaround in https://github.com/jariz/gatsby-plugin-s3/issues/24#issuecomment-565175735 may cause incorrect cache of prefixPath/page-data/**/**.json files. I had to pass in a custom params to workaround this.

When files are moved manually, their paths no longer match https://github.com/jariz/gatsby-plugin-s3/blob/91a012b88aa402142cb8397f4a7e8a8606b727d6/src/constants.ts#L99-L101

Is there a workaround for this? I have set:

module.exports = {
    pathPrefix: `/venues`,
    ...
};

Then run gatsby build --prefix-paths and finally gatsby-plugin-s3 deploy --yes but still uploads all content in the root of the bucket. 😖

Another seeming inconsistency with prefixPath: https://github.com/jariz/gatsby-plugin-s3/blob/40516b39a308ccb337b853a36436beaedca86786/src/bin.ts#L196-L198 sets a non-configurable unprefixed key for 404.html.

My current workaround is to:

yarn run gatsby build --prefix-paths
mv public public-temp
mkdir public
mv public-temp public/OUR_PREFIX
cp -rv public/OUR_PREFIX/404.html public/404.html
yarn gatsby-plugin-s3 deploy

BTW is anyone already working on a fix of pathPrefix support? I might be able to do something🤔

After a survey on previous work of pathPrefix support, I got a little confused.

There was a (perhaps wip) branch at https://github.com/jariz/gatsby-plugin-s3/compare/a0917f..feature//prefix authored by @jariz , but it was not merged into master. And no other commit messages mentioned "pathPrefix" "prefix" or alike.

Is this branch meant for this feature? (it would take some effort to port code and resolve conflicts)

the workaround is causing issues in case there are already other items in the s3 bucket as they get deleted

Is there a workaround for this? I have set:

module.exports = {
    pathPrefix: `/venues`,
    ...
};

Then run gatsby build --prefix-paths and finally gatsby-plugin-s3 deploy --yes but still uploads all content in the root of the bucket. 😖

If my thinking is correct, --prefix-paths is just a rule that actually not a folder, and then when run "gatsby-plugin-s3 deploy" it actually clear your bucket so your local public folder's content uploaded to your root bucket......

any new update? i am still facing the same problem :(

Sorry for the long silence on this issue. I've looked at it several times, but the history is confusing and I haven't had the time to properly untangle it.

As jokester said, jariz worked on this in the feature/prefix branch. It appears jariz mistakenly closed this issue when the branch had not yet been merged.

feature/prefix is now out of date, and none of the project maintainers have the bandwidth to work on this issue at the moment. If someone in the community wants to pick up the torch on Jari's branch, I would gladly review a PR.

A few notes:

• It seems to me that different people in this thread want different things. The original request wants to upload the site to S3 with a prefix, but still display the site at the root of the domain (for example, using CloudFront). But other people are mentioning gatsby's own pathPrefix which tells Gatsby to expect the site to be served from a non-root path on the domain. I'm not really sure that supporting pathPrefix requires anything from us, but if it does please feel free to open a new issue. This issue specifically relates to applying a prefix to the objects we upload to S3. (We should NOT use the pathPrefix option provided by Gatsby for this, as that's likely to cause all sorts of issues.)

• Jari's original work on this predates our updated E2E testing suite. In order for us to merge this now we would need to see it covered by E2E tests.

This issue specifically relates to applying a prefix to the objects we upload to S3

Yep, that's what it is about. Typical use case: Stiching together documentation sites for use with the apollo docs theme. Every sub site is a distinct gatsby site, built under a distinct gatsby path prefix. To deploy all subsites to a single bucket, you also need to be able to add an S3 path prefix for uploads (the same as in the build step in this case).

Btw.: This seems to work now - or do i miss something?