copy directories/files per function

[jamesdixon]

This is a (Bug Report / Feature Proposal)

Question

Description

Hi, is there a way to copy additional directories on a per-function basis? For example, I have a createEmail function located at lib/createEmail that also has a templates directory. I need that copied with the function, but it doesn't copy. However, I also don't want that directory copied to other functions that live in the same project.

Thank you!

For bug reports:

• What went wrong?
• What did you expect should have happened?
• What was the config you used?
• What stacktrace or error message from your provider did you see?

For feature proposals:

• What is the use case that should be solved. The more detail you describe this in the easier it is to understand for us.
• If there is additional config how would it look

Similar or dependent issue(s):

• #12345

Additional Data

• Serverless-Webpack Version you're using: 5.2.0
• Webpack version you're using: 4.16
• Serverless Framework Version you're using: 1.28
• Operating System: OSX High Sierra
• Stack Trace (if available):

[jamesdixon]

@HyperBrain I just tried copy-webpack-plugin in accordance with the advice of #361, but I'm not sure to get specific files to be included in the package of specific functions rather than being copied to every function. Any suggestions?

[HyperBrain]

@jamesdixon This is currently not supported. With the CopyWebpack plugin you'll get the modules copy for each function. @serverless-heaven/serverless-webpack-contributors Any idea, if this can be somehow worked around with existing webpack plugins?

[ceilfors]

I have been using my own written workaround to handle such scenario. See ConditionalPlugin below. Any suggestion on how we can make this as serverless-webpack feature?

const ConditionalPlugin = (condition, plugin) => ({
  apply: compiler => {
    compiler.plugin('emit', (compilation, callback) => {
      if (condition(compiler)) {
        plugin.apply(compiler);
      }
      callback();
    });
  }
});

Inside webpack.config.js used by serverless-webpack:

...
  plugins: [
    ConditionalPlugin(
      compiler => compiler.outputPath.includes('moduleName'), // not function name
      new CopyWebpackPlugin([
        {
          from: 'source directory',
          to: 'lib'
        }
      ])
    )
  ]
...

[jamesdixon]

@ceilfors thanks for this! much appreciated.

I'm running into the following:

/Users/jamesdixon/Projects/scout/platform/functions/email-service/node_modules/copy-webpack-plugin/dist/index.js:163
                for (var _iterator = fileDependencies[Symbol.iterator](), _step; !(_iteratorNormalCompletion = (_step = _iterator.next()).done); _iteratorNormalCompletion = true) {
                                                     ^
TypeError: Cannot read property 'Symbol(Symbol.iterator)' of undefined
    at afterEmit (/Users/jamesdixon/Projects/scout/platform/functions/email-service/node_modules/copy-webpack-plugin/dist/index.js:163:54)
    at AsyncSeriesHook.eval [as callAsync] (eval at create (/Users/jamesdixon/Projects/scout/platform/functions/email-service/node_modules/tapable/lib/HookCodeFactory.js:24:12), <anonymous>:7:1)
    at AsyncSeriesHook.lazyCompileHook [as _callAsync] (/Users/jamesdixon/Projects/scout/platform/functions/email-service/node_modules/tapable/lib/Hook.js:35:21)
    at asyncLib.forEach.err (/Users/jamesdixon/Projects/scout/platform/functions/email-service/node_modules/webpack/lib/Compiler.js:355:27)
    at done (/Users/jamesdixon/Projects/scout/platform/functions/email-service/node_modules/neo-async/async.js:2854:11)
    at /Users/jamesdixon/Projects/scout/platform/functions/email-service/node_modules/neo-async/async.js:2805:7
    at /Users/jamesdixon/Projects/scout/platform/functions/email-service/node_modules/graceful-fs/graceful-fs.js:43:10
    at /Users/jamesdixon/.nvm/versions/node/v8.10.0/lib/node_modules/serverless/node_modules/graceful-fs/graceful-fs.js:43:10
    at FSReqWrap.oncomplete (fs.js:135:15)

Have you had a similar issue?

[ceilfors]

@jamesdixon Unfortunately not. Might be able to help you if you can produce an MVCE.

[jamesdixon]

@ceilfors here's my webpack.config.js for reference:

const nodeExternals = require('webpack-node-externals')
const slsw = require('serverless-webpack')
const CopyWebpackPlugin = require('copy-webpack-plugin')

const ConditionalPlugin = (condition, plugin) => ({
  apply: compiler => {
    compiler.plugin('emit', (compilation, callback) => {
      if (condition(compiler)) {
        plugin.apply(compiler)
      }
      callback()
    })
  }
})

module.exports = {
  entry: slsw.lib.entries,
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  target: 'node',
  devtool: 'source-map',
  optimization: {
    minimize: false
  },
  externals: [nodeExternals()],
  plugins: [
    ConditionalPlugin(
      compiler => compiler.outputPath.includes('createEmail'),
      new CopyWebpackPlugin([
        {
          context: `./lib/createEmail`,
          from: '**/*',
          force: true
        }
      ], {
        copyUnmodified: true
      })
    )
  ]
}

[HyperBrain]

@ceilfors Cool stuff. Maybe the conditional plugin can be integrated into the sls-webpack project, so that you do not need to install it but just use slsw.plugins.ConditionalPlugin in the webpack config?

[hiddestokvis]

@jamesdixon I had the same error, got it working however by using the following code:

const ConditionalPlugin = (condition, plugin) => ({
  apply: compiler => {
    if (condition(compiler)) {
      plugin.apply(compiler)
    }
  }
})

also as a condition I am searching the path compiler.options.output.path instead of compiler.outputPath hope this helps 😄

[jamesdixon]

@hiddestokvis thank you! this fixed my issue 👍

Question: do you have local modules as well? I'm now noticing that paths are off because of the way functions are packaged individually. For example, my functions live under lib/[functionName], so when they are packaged, I end up with that directory structure in the zip file. However, I noticed that no local modules (ex: helpers/) are packaged.

[craigtsmith]

I took the ideas in this thread and extended them so I can set conditionals / config back at the service config level.

Given a serverless.yml with a function like this:

# serverless.yml

...
  handleWebhook:
    handler: handlers/handleWebhook.default
    name: ${self:provider.stage}-${self:service}-handleWebhook
    description: Handle the Webhook
    webpack:
      toggle: true
...

And the conditional plugin tweaked to do this:

const ConditionalPlugin = (condition, plugin) => ({
  apply: compiler => {
    let name = Object.keys(compiler.options.entry)[0].split('/').pop();
    let config = Object.assign({webpack: {}}, slsw.lib.serverless.service.getFunction(name));
    
    if (condition(config)) {
      plugin.apply(compiler)
    }
  }
});

You can configure your functions to have conditional plugins execute based on the configuration in the overall service, like this:

# webpack.config.js

module.exports = {
  entry: slsw.lib.entries,
  target: 'node',
  mode: 'production',
  plugins: [
    ConditionalPlugin(
      ((config) => config.webpack.toggle),
      new CopyWebpackPlugin([{from: 'some/path', to: 'some/other/path'}])
    )
  ],

The only thing specific to my setup is a naming convention, where the function name to serverless is the same as the file name with the lambda handler inside a folder called "handlers".

It seems to be working fine, but YMMV

[buithaibinh]

My solution.

 # serverless.yml
transform:
    handler: src/handler.transform

# webpack.config.js
const ConditionalPlugin = (condition, plugin) => ({
  apply: compiler => {
    if (condition(compiler)) {
      plugin.apply(compiler)
    }
  }
});
...
plugins: [
      ConditionalPlugin(
        compiler => {
          // copy folder templates into webpack
          return compiler.options.output.path.includes("transform")
        },
        new CopyWebpackPlugin([{from:'./templates', to:'templates'} ], {logLevel: "error"}),
      ),
],

[Omicron7]

I wanted to use the packaging config that AWS serverless provides. https://serverless.com/framework/docs/providers/aws/guide/packaging/ Thanks to everyone above, I came up with this.

Individual functions can have

# serverless.yml
functions:
  name:
    handler: path/to/file.handler
    package:
      include:
        - lib/**

I implemented this functionality with

// webpack.config.js
const { find, get } = require('lodash')

module.exports = {
  ...
  plugins: [{
      apply: compiler => {
        const handler = `${Object.keys(compiler.options.entry)[0]}.handler`
        const config = find(slsw.lib.serverless.service.functions, val => val.handler === handler)
        const includePaths = get(config, 'package.include', [])
        if (includePaths.length) {
          new CopyWebpackPlugin(includePaths).apply(compiler)
        }
      }
  }]
}

This takes the array provided to the package.include and sends it to the CopyWebpackPlugin. If anyone knows a better way to find the serverless function config based on the current entrypoint, I'd be grateful. The code above relies on the fact that all of my entrypoints have only a single exported handler name handler.

[cpconcertage]

As someone who treads carefully with webpack, I was thrilled that the solution suggested by @Omicron7 works brilliantly. For some reason (version differences, perhaps), I needed to send the plugin an object with patterns:

new CopyWebpackPlugin({ patterns: includePaths }).apply(compiler);

I don't find it any trouble to call my handler handler, so this solution is a quick and simple fix. Thanks!

[raymond-w-ko]

Minor hack/modification to fix cases where you are specifying a certain file and need to keep the directories. With @Omicron7 's code, if you have:

  function:
    handler: function.handler
    package:
      include:
        - bin/exe

exe gets copied to the root, and it is not inside a ./bin directory in the deployed zip. Below code keeps the ./bin/.

const convertPattern = (s) => {
  if (s.endsWith("*")) return s
  if (!s.includes("/")) return s

  const i = s.lastIndexOf("/")
  return {from: s, to: s.substring(0, i) + "/"}
}

module.exports = {
  plugins: [
    {
      apply: (compiler) => {
        const handler = `${Object.keys(compiler.options.entry)[0]}.handler`
        const config = find(
          slwp.lib.serverless.service.functions,
          (val) => val.handler === handler
        )
        let includePaths = get(config, "package.include", [])
        includePaths = _.map(includePaths, convertPattern)
        if (includePaths.length) {
          new CopyWebpackPlugin({patterns: includePaths}).apply(compiler)
        }
      },
    },
  ],
}

[bboure]

Thank you all for your solutions, They really helped! I'd like to add mine that keeps compatibility with serverless-offline.

plugins: [
    {
      apply: compiler => {
        const handlers = _.map(compiler.options.entry, (val, handler) => {
          return `${handler}.handler`;
        });
        let includePaths = _.flatten(
          _.map(slsw.lib.serverless.service.functions, func => {
            if (_.find(handlers, f => f === func.handler)) {
              return _.get(func, 'package.include', []);
            }

            return [];
          }),
        );
        includePaths = _.map(includePaths, convertPattern);
        if (includePaths.length) {
          new CopyPlugin({ patterns: includePaths }).apply(compiler);
        }
      },
    },
  ],

[arackaf]

A very sincere, heartfelt thank you to everyone contributing solutions here. OSS at its finest. To anyone happening across this thread, I went with a slight variation on this theme

const ConditionalPlugin = (condition, plugin) => ({
  apply: compiler => {
    let name = Object.keys(compiler.options.entry)[0].split('/').pop();
    let config = Object.assign({webpack: {}}, slsw.lib.serverless.service.getFunction(name));
    
    if (condition(config)) {
      plugin.apply(compiler)
    }
  }
});

The only potential problem here is that it assumes the entry file is the same as the function name. If it's not, this will error out. That said, the solution is extremely palatable, especially if you want to enable / disable plugins for all functions in the same source file (which I do)

const ConditionalPlugin = (condition, plugin) => ({
  apply: compiler => {
    let fileName = Object.keys(compiler.options.entry)[0].split("/").pop();

    if (condition(fileName)) {
      plugin.apply(compiler);
    }
  }
});

Which can be used like this

plugins: [
  ConditionalPlugin(
    fileName => fileName != "ws-connection",
    new CopyPlugin({
      patterns: [
        { from: "../node_modules/saslprep", to: "node_modules/saslprep" },
        { from: "../node_modules/sparse-bitfield", to: "node_modules/sparse-bitfield" },
        { from: "../node_modules/memory-pager", to: "node_modules/memory-pager" }
      ]
    })
  )
]

"ws-connection" is my filename, and this invocation adds the CopyPlugin for all functions outside of this file.

Re. the question of baking this into Serverless, I'd argue against that. This is core webpack configuration (if documented extremely poorly - seriously, if someone knows where in the webpack docs this is documented, would you mind commenting?) and so probably should not be bloating up serverless-webpack with a friendly wrapper. imo ymmv.