Package size exceeds AWS Lambda max. deploy size (250MB)

[andreastepel]

Hey there!

I have tried to run the timecut package on AWS Lambda but unfortunately if timecut is installed via npm all dependencies combined result in a deploy package over 300MB, AWS Lambda currently allows only unzipped deploy packages below 250MB. The biggest issue is the Chrome package that is installed by default.

Is there kind of a workaround to maybe cut off some dependencies that are not needed for the core functionalities of timecut? Or has anyone experiences with using timecut on AWS Lambda and has managed to use a package like https://github.com/alixaxel/chrome-aws-lambda ?

Hope that someone has a smart solution for this. Best Andreas

[tungs]

Hi @andreastepel, thanks for this question! It's good to know about the different use cases. As you pointed out, timecut installs timesnap, which installs puppeteer, which installs Chromium, and that takes up a significant amount of space. I don't know if there's a super simple solution for now, though there might be in the future.

In the Future:

The particular case of wanting to use a specific version of puppeteer (in this case one that uses chrome-aws-lambda) is partially the motivation for creating timeweb, which is just the time handling code of timesnap that runs in browser. Because it can be complicated to integrate, I'm planning on eventually making a template repo that integrates it with puppeteer, though that will take some time.

This issue also makes me think it might be useful to have something like timesnap-core and timecut-core that the user can pass a puppeteer-like browser instance.

For Now: Forking

For the time being, you may have some luck forking timesnap and removing puppeteer altogether by changing the lines in index.js/getBrowser() to just return config.browser, removing const puppeteer = require('puppeteer');, and removing puppeteer from the project. Then fork timecut, and have it use this updated version of timesnap. To use this version of timecut, it would then look something like:

const timecut = require('./local/timecut');
const chromium = require('chrome-aws-lambda');

exports.handler = async (event, context, callback) => {
  await timecut({
    browser: chromium.puppeteer.launch({
      args: chromium.args,
      //... the rest of the arguments found in https://github.com/alixaxel/chrome-aws-lambda/blob/master/README.md
    }),
    url: event.url
    // ...
  });
  //
};

Alternatively: Possibly Overriding Dependencies

If you don't want to fork so many things, you might have some luck checking out the answers to this question for overriding dependencies and combining it with aliasing. You might be able to use a custom version of puppeteer via something like "dependencies": { "puppeteer": { "version": "npm:custom-puppeteer-lamda" }} (or "version": "file:./custom-puppeteer", if it's local).

You would either need to find or make a package that mimics the puppeteer API for puppeteer.launch (and maybe puppeteer.connect, if you're using it) for AWS Lamda, though it should be fairly straightforward with chrome-aws-lambda.

I haven't tried any of this stuff, so I have no idea if it works, particularly if npm/yarn can alias packages that way.

[andreastepel]

Hey @tungs!

Thanks for your detailed answer. The idea of making a timecut-core package sounds as good solution. It could just be used as puppeteer-core. The idea of timeweb sounds interesting as well, I will have an eye on the repo.

I will try the approaches you have suggested and will let you know which one worked for me. To find a solution for using timecut on a AWS Lambda function will be a great step to integrate video rendering (e.g. for user-generated content) into many applications on a scalable and affordable basis.

[tungs]

Hi @andreastepel, as previously discussed, I just published timesnap-core and timecut-core, which doesn't include the puppeteer dependencies.

You can use it by via:

npm install timecut-core --save

And then in your program, the implementation would look something like this:

const timecut = require('timecut-core');
const chromium = require('chrome-aws-lambda');

exports.handler = async (event, context, callback) => {
  await timecut({
    browser: chromium.puppeteer.launch({
      args: chromium.args,
      //... the rest of the arguments found in https://github.com/alixaxel/chrome-aws-lambda/blob/master/README.md
    }),
    url: event.url
    // ...
  });
  //
};

Thanks for your interest in this! Let me know if it works or if you run into any problems.

[andreastepel]

Hey @tungs!

Thank you for the nice solution! I have tried it and it works very well 🎉

It has become very easy now to render Web Animations on a scalable infrastructure.

Some hints for everybody who wants to try this:

• you have to set frameCache: '/tmp' and output: '/tmp' because AWS Lambda only allows you to write inside the /tmp directory
• to get FFMPEG running you have to add a custom layer which installs FFMPEG for your Lambda function (some hints how this works can be found in this post in section 2)

I have made some performance tests as well for everybody who is interested:

• Animation was 15sec with 1920x1080px resolution and 24fps framerate
• screenshotType: 'jpeg' was used to speed up performance
• 1024MB RAM -> 1min 34sec
• 2048MB RAM -> 56sec
• 4096MB RAM -> 47sec
• 8192MB RAM -> 35sec

Hope that helps anybody who is running into the same question.

[Asheboy]

Thanks for those performance tests, very useful!

On Wed, 27 Jan 2021 at 15:14, Andreas Tepel [email protected] wrote:

Hey @tungs https://github.com/tungs!

Thank you for the nice solution! I have tried it and it works very well 🎉

It has become very easy now to render Web Animations on a scalable infrastructure.

Some hints for everybody who wants to try this:

• you have to set frameCache: '/tmp' and output: '/tmp' because AWS Lambda only allows you to write inside the /tmp directory
• to get FFMPEG running you have to add a custom layer which installs FFMPEG for your Lambda function (some hints how this works can be found in this post https://aws.amazon.com/de/blogs/media/processing-user-generated-content-using-aws-lambda-and-ffmpeg/ in section 2

I have made some performance tests as well for everybody who is interested:

• Animation was 15sec with 1920x1080px resolution and 24fps framerate
• screenshotType: 'jpeg' was used to speed up performance
• 1024MB RAM -> 1min 34sec
• 2048MB RAM -> 56sec
• 4096MB RAM -> 47sec
• 8192MB RAM -> 35sec

Hope that helps anybody who is running into the same question.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tungs/timecut/issues/44#issuecomment-768353555, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAN46QNRGGDNK3JKSXFL4CTS4AUWHANCNFSM4WIOE2RQ .

--

[email protected] Senior Software Engineer

Office: +44 1923 261166

--

• http://www.clock.co.uk/

https://www.clock.co.uk/insight/clock-launches-two-new-sites-for-BBFC

_ _ The Old School House Bridge Road, Hunton Bridge, Kings Langley, WD4 8RQ

Switchboard: +44 1923 261166 www.clock.co.uk http://www.clock.co.uk/

• http://twitter.com/clock    https://github.com/clocklimited    http://www.linkedin.com/company/clock-limited   

www.clock.co.uk/email-disclaimer http://www.clock.co.uk/email-disclaimer clock.® is a Registered Trademark of Clock Limited in the European Community and other territories

[tdurand]

@andreastepel I'm two years later and I'm getting much lower speeds on AWS Lambda (1024 MB), using a newer version of puppeteer and chrome, with screenshotType = "jpeg" and mode canvas.. for 1920x1080 px resolution, it takes forever: like 1 min for 1 second of animation at 30 fps.. the screenshot speed is something like 1 per second...

Any tips ? I will maybe try to downgrade node / chromium / puppeteer

[tdurand]

downgrading to node v12 and "chrome-aws-lambda": "^5.5.0" "puppeteer-core": "^5.5.0" gives a bit better results:

for a 5s animation 30 FPS 1920x1080 (150 frames) jpeg mode

• 1024 MB -> 1min40
• 2048 MB -> 1 min

Still like 3 times slower than @andreastepel though

[WofWca]

Thanks for the tests @andreastepel and @tdurand ! Do you also have price estimates handy by any chance?