[email protected]

[Sparticuz]

This PR updates the build process for chromium and gets puppeteer support up to 14.1.0.

Closes: #240 #248 #258 #254 #242 #274 #275 #276

Notes:

• Ignore the branch name, I originally created the PR for Puppeteer 13.5.0
• I am not going to include the binaries in this PR. If you would like to use my binaries or my lambda layer, please see my latest comments
• If you would like to use a scoped npm package that will be kept up to date until this PR is merged, check out @sparticuz/chrome-aws-lambda

Due to some changes in WebGL/SWAngle, this PR does not included the needed fixes for lambdafs. If you need WebGL, please use my scoped package.

[deldrid1]

@Sparticuz - do you mind to commit your bin/chromium.br and bin/swiftshader.tar.br? I'd love to test this in my build environment with Typescript and webpack but don't have the bandwidth to spin up the compilation side at the moment...

[Sparticuz]

After cleaning everything up, I'm receiving the same WebGL error's that @alixaxel was getting. (I'm unsure why my initial compile seemed to work, but I haven't been able to reproduce that). It looks like in Chrome 96, swiftshader was dropped and replaced with swangle, so I'm doing some research into what needs to be done to switch to that. I even got vulkan working on my local box and got better performance, but I'm unsure if that would work on lambda or not.

[Sparticuz]

I believe I've found the files needed to be bundled for Chrome 96+.

• libEGL.so
• libGLESv2.so
• libvk_swiftshader.so
• libvulkan.so.1
• vk_swiftshader_icd.json
• (obviously headless_shell as chromium as well)

Notice that the swiftshader/libEGL.so and swiftshader/libGLESv2.so are not included.

Also the cli flags that needed to be changed are:

• --use-gl=angle
• --use-angle=swiftshader

With those files and those flags, chromium runs without error and an inspect shows a working webgl.

I'm running a compile with the latest ansible instructions in order to bundle the needed files. I won't commit the brotli files. I'll let @alixaxel run a build for security reasons, but I will post them so they can be tested. The compile takes about an hour. @alixaxel I also won't commit the changes I made to the ec2 instance, but I'll tell you I upgraded to a c6i.12xlarge.

After I test this out I'll push the latest changes as Ready for review

[Sparticuz]

~~https://raw.githubusercontent.com/Sparticuz/chrome-aws-lambda/61139ebc4a62f8a2576ad79be858e13204c010b4/bin/chromium.br~~

~~https://raw.githubusercontent.com/Sparticuz/chrome-aws-lambda/61139ebc4a62f8a2576ad79be858e13204c010b4/bin/swiftshader.tar.br~~

EDIT: If you use these, you'll need to edit chrome-aws-lambda's index file to include the flags in the previous comment.

Edit2: Please see my branch here for the latest binaries https://github.com/Sparticuz/chrome-aws-lambda/tree/13.5_bin/bin

[Sparticuz]

~~Here is a Lambda layer: https://raw.githubusercontent.com/Sparticuz/chrome-aws-lambda/b4584a6c65e1c33d7a1afc1edbb825ee4f2509a6/chrome_aws_lambda.zip~~

I've tested it with the SAM application in the _/amazon folder and it worked. I'll be testing it internally as well.

EDIT: Please see my branch here for the latest layer: https://github.com/Sparticuz/chrome-aws-lambda/blob/13.5_bin/chrome_aws_lambda.zip

[deldrid1]

@Sparticuz - just to let you know, I've merged in with your _bin branch on top of my changes for webpack and so far so good! I'm doing some larger sized testing today, but nothing to complain about with the updates! (I don't require webgl so I can't speak to that, but basic functionality looks good!)

[ibqn]

@Sparticuz great contribution! I am looking forward to see this PR getting approved.

[owlyowl]

@Sparticuz thanks so much for getting this up and running it will be a huge help to the community!

Are there any particular lambda config changes that are required to support the new layer/version?

[bestplay9384]

Hey @Sparticuz,

I'm testing Your compilation on AWS Lambda and with DEBUG enabled i got this on chromium boot.

I have chromium flags you suggested above. Any idea what can it be? :)

EDIT: flag that is not included in chrome-aws-lambda flags - --disable-software-rasterizer made errors disappear :) I thought it is an issue with webGL/driver changes you've made.

[Sparticuz]

It looks like it can't find libglesv2.so. are you using the zip image, or did you build and upload?

[bestplay9384]

It looks like it can't find libglesv2.so. are you using the zip image, or did you build and upload?

I was using brotli compressed archives from Your fork :) Those You have linked above :) I have it all packed to lambda layer and then attached to lambda.

EDIT: I've just found out that my custom decompression code for lambda layer build extracts swiftshader libnraries to bin/swiftshader directory - not in bin itself. It is probably an issue - paths mismatch :)

EDIT 2: Yup, fixed paths, removed --disable-software-rasterizer flag and it works as it should :)

[Sparticuz]

I've updated my PR to be Puppeteer 13.6.0, and my _bin branch to have 13.6.0 as well as chromium 101. Layer here. I have not tested it

[lewisl9029]

I gave the 13.6.0 layer a shot, and first few invocations seem to work fine (albeit quite a bit slower than the version on main for some reason), but afterwards I get errors that look like this when trying to take a screenshot:

2022-04-21T02:06:48.862Z pw:browser [pid=103][err] [0421/020648.861048:ERROR:shared_image_representation.cc(219)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (1280, 256)
2022-04-21T02:06:48.864Z pw:browser [pid=103][err] [0421/020648.864208:ERROR:shared_image_representation.cc(219)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (832, 192)
2022-04-21T02:06:48.864Z pw:browser [pid=103][err] [0421/020648.864616:ERROR:shared_image_representation.cc(219)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (1280, 64)
2022-04-21T02:06:48.930Z pw:browser [pid=103][err] [0421/020648.929945:ERROR:validation_errors.cc(117)] Invalid message: VALIDATION_ERROR_DESERIALIZATION_FAILED
2022-04-21T02:06:48.930Z pw:browser [pid=103][err] [0421/020648.929974:WARNING:network_service.cc(202)] Mojo error in NetworkService: Validation failed for viz.mojom.CopyOutputResultSender.0  [VALIDATION_ERROR_DESERIALIZATION_FAILED]
2022-04-21T02:06:48.930Z pw:browser [pid=103][err] [0421/020648.929983:ERROR:interface_endpoint_client.cc(664)] Message 500872390 rejected by interface viz.mojom.CopyOutputResultSender
2022-04-21T02:06:48.954Z pw:browser [pid=103][err] [0421/020648.954338:ERROR:shared_image_representation.cc(219)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (576, 192)
2022-04-21T02:06:48.955Z pw:browser [pid=103][err] [0421/020648.955260:ERROR:shared_image_manager.cc(227)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
2022-04-21T02:06:48.956Z pw:browser [pid=103][err] [0421/020648.956010:ERROR:shared_image_manager.cc(227)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
2022-04-21T02:06:48.957Z pw:browser [pid=103][err] [0421/020648.957278:ERROR:shared_image_manager.cc(227)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
2022-04-21T02:06:48.963Z pw:browser [pid=103][err] [0421/020648.963574:ERROR:shared_image_manager.cc(227)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
2022-04-21T02:06:48.964Z pw:browser [pid=103][err] [0421/020648.964137:ERROR:shared_image_representation.cc(219)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (1280, 64)
2022-04-21T02:06:49.026Z pw:browser [pid=103][err] [0421/020649.026090:ERROR:validation_errors.cc(117)] Invalid message: VALIDATION_ERROR_DESERIALIZATION_FAILED
2022-04-21T02:06:49.026Z pw:browser [pid=103][err] [0421/020649.026117:WARNING:network_service.cc(202)] Mojo error in NetworkService: Validation failed for viz.mojom.CopyOutputResultSender.0  [VALIDATION_ERROR_DESERIALIZATION_FAILED]
2022-04-21T02:06:49.026Z pw:browser [pid=103][err] [0421/020649.026127:ERROR:interface_endpoint_client.cc(664)] Message 500872390 rejected by interface viz.mojom.CopyOutputResultSender
2022-04-21T02:06:49.047Z	eb6fc018-3b22-4243-b9f3-743a206c7cdc	ERROR	Invoke Error 	{"errorType":"Error","errorMessage":"page.screenshot: Protocol error (Page.captureScreenshot): Unable to capture screenshot\n=========================== logs ===========================\ntaking page screenshot\n============================================================","name":"Error","stack":["page.screenshot: Protocol error (Page.captureScreenshot): Unable to capture screenshot","=========================== logs ===========================","taking page screenshot","============================================================","    at /var/task/main.cjs:174:10","    at measure (/var/task/main.cjs:35:24)","    at Runtime.module.exports.handler (/var/task/main.cjs:173:28)"]}
END RequestId: eb6fc018-3b22-4243-b9f3-743a206c7cdc

FWIW, I'm using playwright instead of puppeteer, but curious if others had success with taking screenshots using this version (puppeteer or playwright)?

[bestplay9384]

Hey @Sparticuz, I've got a chance to test quite much Your changes for 13.5.0/13.5.1. I'm using puppeteer 13.5.1 with chromium 100 (from Your fork). I'm doing around 100k invokes od lambda daily, doing work for me using puppeteer, none of these invokes failed because of chromium/puppeteer issues, so i'm pretty sure it works fine :) I'm just scraping websites so I'm not sure if any of webGL stuff had a chance to occur :/ Just givin' ya all info it works for my needs.

[Sparticuz]

I gave the 13.6.0 layer a shot, and first few invocations seem to work fine (albeit quite a bit slower than the version on main for some reason), but afterwards I get errors that look like this when trying to take a screenshot:

2022-04-21T02:06:48.862Z pw:browser [pid=103][err] [0421/020648.861048:ERROR:shared_image_representation.cc(219)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (1280, 256)
2022-04-21T02:06:48.864Z pw:browser [pid=103][err] [0421/020648.864208:ERROR:shared_image_representation.cc(219)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (832, 192)
2022-04-21T02:06:48.864Z pw:browser [pid=103][err] [0421/020648.864616:ERROR:shared_image_representation.cc(219)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (1280, 64)
2022-04-21T02:06:48.930Z pw:browser [pid=103][err] [0421/020648.929945:ERROR:validation_errors.cc(117)] Invalid message: VALIDATION_ERROR_DESERIALIZATION_FAILED
2022-04-21T02:06:48.930Z pw:browser [pid=103][err] [0421/020648.929974:WARNING:network_service.cc(202)] Mojo error in NetworkService: Validation failed for viz.mojom.CopyOutputResultSender.0  [VALIDATION_ERROR_DESERIALIZATION_FAILED]
2022-04-21T02:06:48.930Z pw:browser [pid=103][err] [0421/020648.929983:ERROR:interface_endpoint_client.cc(664)] Message 500872390 rejected by interface viz.mojom.CopyOutputResultSender
2022-04-21T02:06:48.954Z pw:browser [pid=103][err] [0421/020648.954338:ERROR:shared_image_representation.cc(219)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (576, 192)
2022-04-21T02:06:48.955Z pw:browser [pid=103][err] [0421/020648.955260:ERROR:shared_image_manager.cc(227)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
2022-04-21T02:06:48.956Z pw:browser [pid=103][err] [0421/020648.956010:ERROR:shared_image_manager.cc(227)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
2022-04-21T02:06:48.957Z pw:browser [pid=103][err] [0421/020648.957278:ERROR:shared_image_manager.cc(227)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
2022-04-21T02:06:48.963Z pw:browser [pid=103][err] [0421/020648.963574:ERROR:shared_image_manager.cc(227)] SharedImageManager::ProduceSkia: Trying to Produce a Skia representation from a non-existent mailbox.
2022-04-21T02:06:48.964Z pw:browser [pid=103][err] [0421/020648.964137:ERROR:shared_image_representation.cc(219)] Attempt to read from an uninitialized SharedImage. Initialized region: (0, 0, 0, 0) Size: (1280, 64)
2022-04-21T02:06:49.026Z pw:browser [pid=103][err] [0421/020649.026090:ERROR:validation_errors.cc(117)] Invalid message: VALIDATION_ERROR_DESERIALIZATION_FAILED
2022-04-21T02:06:49.026Z pw:browser [pid=103][err] [0421/020649.026117:WARNING:network_service.cc(202)] Mojo error in NetworkService: Validation failed for viz.mojom.CopyOutputResultSender.0  [VALIDATION_ERROR_DESERIALIZATION_FAILED]
2022-04-21T02:06:49.026Z pw:browser [pid=103][err] [0421/020649.026127:ERROR:interface_endpoint_client.cc(664)] Message 500872390 rejected by interface viz.mojom.CopyOutputResultSender
2022-04-21T02:06:49.047Z	eb6fc018-3b22-4243-b9f3-743a206c7cdc	ERROR	Invoke Error 	{"errorType":"Error","errorMessage":"page.screenshot: Protocol error (Page.captureScreenshot): Unable to capture screenshot\n=========================== logs ===========================\ntaking page screenshot\n============================================================","name":"Error","stack":["page.screenshot: Protocol error (Page.captureScreenshot): Unable to capture screenshot","=========================== logs ===========================","taking page screenshot","============================================================","    at /var/task/main.cjs:174:10","    at measure (/var/task/main.cjs:35:24)","    at Runtime.module.exports.handler (/var/task/main.cjs:173:28)"]}
END RequestId: eb6fc018-3b22-4243-b9f3-743a206c7cdc

FWIW, I'm using playwright instead of puppeteer, but curious if others had success with taking screenshots using this version (puppeteer or playwright)?

Bfcache was disabled in puppeteer 13.6, Try that and see if you are getting the same results https://github.com/puppeteer/puppeteer/issues/8182

[lewisl9029]

A few more updates:

I gave disabling bfcache a try, but unfortunately it didn't change anything. Also tried an earlier version of the layer at https://github.com/alixaxel/chrome-aws-lambda/pull/264#issuecomment-1064166840, and it had the same issue.

Afterwards, I tried migrating the script to puppeteer, which worked (albeit with a screenshot time of ~1s compared to ~200ms with playwright on main)! So this seems to be a playwright specific thing. Going to dig into playwright's default flags to see if there's anything I'm missing that might resolve this.

EDIT: Another update:

Figured it out! Apparently this was caused by my custom pre-inflating layer I was using with the version on main to improve cold starts. Seems like that approach no longer worked with this version for some reason.

Everything started working again once I reverted back to using the standard await chromeAwsLambda.executablePath approach with the layer you provided. Apologies for the false alarm, and thank you so much for working on this!

[ferbs]

I'm using a build based on this fork and it seems to work perfectly. Am running it in a container image based on public.ecr.aws/lambda/nodejs:14. Thank you @sparticuz!

Hard to know when this fork will get merged in, so in case anyone needs it, here are instructions for installing from the fork:

git clone https://github.com/Sparticuz/chrome-aws-lambda.git
# ^^^ note: almost 2GB. Maybe do a depth=1 clone of just the [email protected] branch. (I don't remember that git command though) 

git checkout [email protected]
# ^^^ keep an eye for newer, in case it moves since this was posted. 

npm install
npm pack

# next, replace $your_branch below with actual path:
mv chrome-aws-lambda-13.6.0.tgz $your_branch && cd $your_branch 
npm i --save ./chrome-aws-lambda-13.6.0.tgz

[Sparticuz]

git checkout [email protected]
# ^^^ keep an eye for newer, in case it moves since this was posted. 

I'll keep the branch the same for the purposes of this PR, but I'll update the PR title and comment when there is a new version of Puppeteer that requires a new build of chromium.

Ideally we can get this merged, then get some automated testing on it. I still don't know enough about the typescript that @alixaxel has written to be confident enough to say one way or the other if what I've done is production ready because of the lack of automated testing. There is only the one SAM test, but it's only testing that chromium takes a screenshot. It doesn't test any other functionality (webgl, etc...)

That said, a number of users ARE using it in production.

[ferbs]

@Sparticuz Maybe running the core puppeteer test suite against this build might be a good solution? From a skim of the code, it looks like puppeteer is using an integration test approach, hitting an actual browser for everything. It uses process.env.BINARY to set executablePath (in test/mocha-utils.ts) so plopping their entire repo into a Lambda and running BINARY=$chrome-aws-lambda npm run test might be a decent solution.

[byF]

@Sparticuz I'm getting following Chromium error for both 13.5 and 13.6 layers built by you. It doesn't crash the browser and its functionality seems to be fine (I've got a different problem with fonts, but I'm investigating if it's related or not EDIT: very likely not)

[0427/144243.098580:ERROR:egl_util.cc(74)] Failed to load GLES library: /tmp/libGLESv2.so: /tmp/libGLESv2.so: cannot open shared object file: No such file or directory
[0427/144243.104908:ERROR:gpu_channel_manager.cc(831)] ContextResult::kFatalFailure: Failed to create shared context for virtualization.
[0427/144243.104968:ERROR:shared_image_stub.cc(537)] SharedImageStub: unable to create context
[0427/144243.104976:ERROR:gpu_channel.cc(571)] GpuChannel: Failed to create SharedImageStub
[0427/144243.104999:ERROR:gpu_channel_manager.cc(831)] ContextResult::kFatalFailure: Failed to create shared context for virtualization.
[0427/144243.105004:ERROR:shared_image_stub.cc(537)] SharedImageStub: unable to create context
[0427/144243.105008:ERROR:gpu_channel.cc(571)] GpuChannel: Failed to create SharedImageStub

[Sparticuz]

[0427/144243.098580:ERROR:egl_util.cc(74)] Failed to load GLES library: /tmp/libGLESv2.so: /tmp/libGLESv2.so: cannot open shared object file: No such file or directory

Others have noted that this might happen when using custom decompression. It's basically failing on loading up the WebGL stuff, so if you aren't using webgl, you can ignore that.

[Sparticuz]

I've updated this PR to 13.7.0, and updated my '13.5_bin' branch to include the binaries and the layer.

[brunolemos]

is this on npm somewhere?

[Sparticuz]

I've been contemplating it, but it would be namespaced and deprecated after it's merged. Maybe I'll do that soon. @alixaxel, my preference would be to get this merged.

[Sparticuz]

I've updated this PR to 14.0.0, and updated my '13.5_bin' branch to include the binaries and the layer.

I think I have decided to go ahead and publish a namespaced npm package. If v14 proves to be good, I'll post it later this week.

Note that puppeteer 14 drops node 12 support and includes ESM package support. I didn't update any of the chrome-aws-linux typescript, only updated the chromium binary, so please post if there are issues, I won't have time to update my deployment from 13.7 until later this week or next, so I haven't tested 14 yet.

[yakirza17]

Hi @Sparticuz, can you put the 14 binary files inside a new branch/tag called v14?

Thanks for your updates

[otaviosoares]

I've updated this PR to 14.0.0, and updated my '13.5_bin' branch to include the binaries and the layer.

I tried to use the layer version 14 but got the error Cannot find module '/opt/nodejs/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Browser'. could I be missing something? Thanks

[terebentina]

@Sparticuz using the latest layer you published I get this error:

error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

My serverless layer is defined like this:

layers:
  HeadlessChrome:
    name: HeadlessChrome
    compatibleRuntimes:
      - nodejs16.x
    description: Required for headless chrome
    package:
      artifact: layer.zip

(layer.zip is your layer)

[hmazter]

@terebentina This is not compatible with nodjs16.x as far as I know. There is a PR for that: #274

[terebentina]

Right, I see. Tried with node 14 as well and now I also get the error reported above: Cannot find module '/opt/nodejs/node_modules/puppeteer-core/lib/cjs/puppeteer/common/Browser'

[Sparticuz]

I'm also getting that error, I haven't had time to look into it yet. If anyone finds a solution let me know.

I'll pull in the node16 patch. I'm currently migrating to 16 as well.

[otaviosoares]

Adding the .js extension to the require seems to fix it.

Super = require('puppeteer-core/lib/cjs/puppeteer/common/Browser.js').Browser;

It's probably because puppeteer 14 added an exports block to its package.json.

[Sparticuz]

Adding the .js extension to the require seems to fix it.

Fantastic, I'll get this added asap.

[terebentina]

Sparticuz, the layer has chrome-aws-lambda installed but it should have @Sparticuz/chrome-aws-lambda

[Sparticuz]

Sorry, I was trying to change the branch name and I guess Github closed the PR.

[Sparticuz]

Sparticuz, the layer has chrome-aws-lambda installed but it should have @Sparticuz/chrome-aws-lambda

I'm going to keep this PR using chrome-aws-lambda, but I am prepping to do a scoped package.

[Sparticuz]

Here is a scoped npm package for chrome-aws-lambda. @sparticuz/chrome-aws-lambda

I will continue to keep this PR update to date, but will probably not keep the layer up to date as it can be built via my scoped package.

[tomoima525]

Thank you @Sparticuz it's working with Node v16 environment on AWS Lambda!

[Antoine-C]

Thanks for the hard work @Sparticuz, I'm wondering if anyone can access https://get.webgl.org/ with a working "spinning cube" ? On my end with your binaries I only have the following error message "Hmm. While your browser seems to support WebGL, it is disabled or unavailable. If possible, please ensure that you are running the latest drivers for your video card."

With https://browserleaks.com/webgl I can see, for the WebGL Support Detection, the following :

• This browser supports WebGL : False (supported, but disabled in browser settings, or blocked by extensions).
• This browser supports WebGL 2 : False (supported, but disabled in browser settings, or blocked by extensions).

I've been looking at https://peter.sh/experiments/chromium-command-line-switches/ but can't see an option that could cause this. I've set --disable-webgl=false , --disable-webgl-image-chromium=false and --disable-webgl2=false for the chromium.args but without luck.

I know WebGL is rendering when using the --use-gl=swiftshader but as soon as I swtich to --use-gl=andle and --use-angle=swiftshader it's not working anymore.

Edit : Also what's weird is this mailchimp webgl website works while webgl.org doesn't.

Do you have any idea ?

[Sparticuz]

@Antoine-C I would double check your code and verify that you are running my fork, or this PR. It's working on my end.

[Antoine-C]

@Antoine-C I would double check your code and verify that you are running my fork, or this PR. It's working on my end.

Thanks a lot for checking on this one, I'll double check everything I must have something wrong somewhere.

[otisg]

Just scanned the responses here and it looks like @alixaxel is not really maintaining this project. No comments from him in this PR (since March 8, 2022) and no merged PRs since last year (since September 2021).

Would it make sense to abandon this project and for people to switch to using https://www.npmjs.com/package/@sparticuz/chrome-aws-lambda instead? @Sparticuz - do you have interest in maintaining the project on an ongoing basis?

[Sparticuz]

@otisg I plan on maintaining my fork, (as well as this PR). I also see no need to change from the established api, so it should also be an easy drop in replacement.

I would love to see some PRs with more automated testing. I think this is one of the main reasons devs lose the ability to maintain a package. The only tests that are currently run test that chromium opens and goes to a website. None of the hooks or puppeteer overrides are tested. Check out _/amazon/handlers/index.js. Personally, I like ava and the node18 testing api seems extremely similar when we are able to update

[yakirza17]

Hi, @Sparticuz, thanks for your great work! I'm trying to use the binary files you are published but cannot make WebGL work.

I can see this on stderr: [0522/162950.046166:ERROR:gpu_channel_manager.cc(831)] ContextResult::kFatalFailure: Failed to create shared context for virtualization. [0522/162950.046194:ERROR:shared_image_stub.cc(537)] SharedImageStub: unable to create context [0522/162950.046203:ERROR:gpu_channel.cc(572)] GpuChannel: Failed to create SharedImageStub

And in get.webgl.org I see that my browser doesn't support WebGL.

There is some configuration that we need to do to make this works inside regular docker lambda? (We are using Node 14)

More details:

• We added these flags to the browser: '--use-gl=angle', // https://chromium.googlesource.com/chromium/src/+/main/docs/gpu/swiftshader.md '--use-angle=swiftshader', // https://chromium.googlesource.com/chromium/src/+/main/docs/gpu/swiftshader.md

• We build dokcer file from this template FROM public.ecr.aws/lambda/nodejs:14-x86_64

• In older versions of this package version, all works, we just want to upgrade our version.

[Antoine-C]

@Sparticuz Can confirm the issue with WebGL as @yakirza17 mentioned, double checked everything, used your fork with binaries and webgl is not working using the official lambda docker image FROM amazon/aws-lambda-nodejs:14 I also tried --use-angle=swiftshader-webgl without luck.

[Sparticuz]

@yakirza17 @Antoine-C I'm also able to confirm that webgl isn't working properly inside of lambda. Previously, I had only tested the chromium binaries directly (in WSL2) and never tested them (with webgl) inside lambda. (If you don't need webgl, my fork and PR work fine)

I've started working on trying to get some automated webgl testing, and I've come to the conclusion that the binaries are extracting incorrectly inside of /tmp. (Where chrome-aws-lambda expects them to be). All the swiftshader files are extracted to /tmp/swiftshader/* where I need them to be in /tmp/*. This looks like it might need to be addressed in @alixaxel/lambdafs, but I'm looking into it.

[yakirza17]

@Sparticuz Extract swiftshader.tar.br into the main directory instead of /swiftshader directory solves the issue.

Thank you very much

[Sparticuz]

I've published 14.1.1 and I believe I've addressed the webgl problem. @Antoine-C @yakirza17 Let me know if it works and I'll put my updates in this PR.

I had to 'fork' lambdafs in order to get the files. Check out source/lambdafs.ts https://github.com/Sparticuz/chrome-aws-lambda/blob/master/source/lambdafs.ts. (instead of forking, I've just moved the relevant code into this package and dropped the dependency). The main change was if we are extracting swiftshader.tar.bz then we use /tmp as the output path instead of /tmp/__filename__. This puts the files in the same basedir as the chromium executable.

[aeristhy]

does this fix this issue #276 ?

[Sparticuz]

does this fix this issue #276 ?

Yes

[aeristhy]

@Sparticuz Hi there, is there anyway to implement this on my codes/dep without manually copy pasting? Should i make a private npm package using this pr?

[Sparticuz]

@aeristhy

If you would like to use a scoped npm package that will be kept up to date until this PR is merged, check out @sparticuz/chrome-aws-lambda

[bjassael]

@Sparticuz thanks for the PR. After updating my code to use these changes there is a drop in performance when taking screenshots with Puppeteer in AWS (from '< 1' to '~8' seconds). Also, total memory consumption almost double from ~450mb to ~800mb. Any clue of what could have trigger this problem or possible fixes?

[Sparticuz]

After updating my code to use these changes there is a drop in performance

Are you using this PR or my fork? This branch does not have the updated swiftshader code, so chromium is likely searching for the right files and not finding them. I believe someone further up had similar problems.

I'm still trying to figure out how to update this PR with the needed changes.

My fork includes the needed changes to get webgl working.

That would be my guess, but I haven't measured anything.

[bjassael]

Sorry, I didn't specified, I'm using the fork. So it does have the webGL changes.

[codykaup]

Thanks for putting this together! I tried using the binary from your v14.3.0 branch with Playwright v1.22.2 but I keep getting the following error whenever I try to create a new page:

[err] [ERROR:file_path_watcher_inotify.cc(329)] inotify_init() failed: Function not implemented (38)
[err] Received signal 11 SEGV_MAPERR XXXXXXXXXXXXX

Is anybody else hitting this? Am I doing something wrong?

[Antoine-C]

@codykaup I had the same error when using Lambda docker image on Apple M1, you need to add the --platform linux/amd64 in order to get rid of this error 👍

[Antoine-C]

@Sparticuz I can confirm the issue @bjassael is having with performance degradation with the WebGL fix, I can see screenshots being 10x slower (I see numbers going from 150ms to 1.2s+). Maybe using angle with swiftshader is causing this degradation given without the webgl fix it was not using swiftshader libs right ?

[codykaup]

Thanks for the heads up @Antoine-C, it's working locally now!

I tried running this in Lambda but it throws a different error for some reason. I tried multiple versions (v14.1.0, v14.1.1, and v14.3.0) but all of them throw this same error. Any ideas on why the binaries from @Sparticuz's fork does this with Playwright?

browser.newPage: Browser closed.
==================== Browser output: ====================
<launching> /tmp/chromium --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --headless --hide-scrollbars --mute-audio --blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4 --no-sandbox --allow-running-insecure-content --autoplay-policy=user-gesture-required --disable-component-update --disable-domain-reliability --disable-features=AudioServiceOutOfProcess,IsolateOrigins,site-per-process --disable-print-preview --disable-setuid-sandbox --disable-site-isolation-trials --disable-speech-api --disable-web-security --disk-cache-size=33554432 --enable-features=SharedArrayBuffer --hide-scrollbars --ignore-gpu-blocklist --in-process-gpu --mute-audio --no-default-browser-check --no-pings --no-sandbox --no-zygote --use-gl=angle --use-angle=swiftshader --window-size=1920,1080 --single-process --font-render-hinting=slight --user-data-dir=/tmp/playwright_chromiumdev_profile-W8GLWJ --remote-debugging-pipe --no-startup-window
<launched> pid=476
[pid=476][err] Received signal 11 SEGV_MAPERR 0000000000b0
[pid=476][err] #0 0x5604b9191cb9 (/tmp/chromium+0x5938cb8)
[pid=476][err] #1 0x5604b9118af3 (/tmp/chromium+0x58bfaf2)
[pid=476][err] #2 0x5604b91917c1 (/tmp/chromium+0x59387c0)
[pid=476][err] #3 0x7fc617d898e0 (/usr/lib64/libpthread-2.26.so+0x118df)
[pid=476][err] #4 0x5604b7d6610a (/tmp/chromium+0x450d109)
[pid=476][err] #5 0x5604b7d65189 (/tmp/chromium+0x450c188)
[pid=476][err] #6 0x5604b7d9fb14 (/tmp/chromium+0x4546b13)
[pid=476][err] #7 0x5604b7d9f90e (/tmp/chromium+0x454690d)
[pid=476][err] #8 0x5604b7cbf8df (/tmp/chromium+0x44668de)
[pid=476][err] #9 0x5604b7ed846d (/tmp/chromium+0x467f46c)
[pid=476][err] #10 0x5604b7ecc5f6 (/tmp/chromium+0x46735f5)
[pid=476][err] #11 0x5604b90f8f9e (/tmp/chromium+0x589ff9d)
[pid=476][err] #12 0x5604bcdfcb18 (/tmp/chromium+0x95a3b17)
[pid=476][err] #13 0x5604bce06ce3 (/tmp/chromium+0x95adce2)
[pid=476][err] #14 0x5604bce06e49 (/tmp/chromium+0x95ade48)
[pid=476][err] #15 0x5604bce0d0c9 (/tmp/chromium+0x95b40c8)
[pid=476][err] #16 0x5604b8978a03 (/tmp/chromium+0x511fa02)
[pid=476][err] #17 0x5604bce04418 (/tmp/chromium+0x95ab417)
[pid=476][err] #18 0x5604bce020b8 (/tmp/chromium+0x95a90b7)
[pid=476][err] #19 0x5604b7a5b98e (/tmp/chromium+0x420298d)
[pid=476][err] #20 0x5604b7a5b37a (/tmp/chromium+0x4202379)
[pid=476][err] #21 0x5604b7a57622 (/tmp/chromium+0x41fe621)
[pid=476][err] #22 0x5604b9151c50 (/tmp/chromium+0x58f8c4f)
[pid=476][err] #23 0x5604b9165a20 (/tmp/chromium+0x590ca1f)
[pid=476][err] #24 0x5604b916572c (/tmp/chromium+0x590c72b)
[pid=476][err] #25 0x5604b9165fa2 (/tmp/chromium+0x590cfa1)
[pid=476][err] #26 0x5604b91aefcb (/tmp/chromium+0x5955fca)
[pid=476][err] #27 0x5604b9166226 (/tmp/chromium+0x590d225)
[pid=476][err] #28 0x5604b91391ed (/tmp/chromium+0x58e01ec)
[pid=476][err] #29 0x5604b79d588d (/tmp/chromium+0x417c88c)
[pid=476][err] #30 0x5604b79d6ed2 (/tmp/chromium+0x417ded1)
[pid=476][err] #31 0x5604b90fc4ce (/tmp/chromium+0x58a34cd)
[pid=476][err] #32 0x5604b812adb2 (/tmp/chromium+0x48d1db1)
[pid=476][err] #33 0x5604b812bfde (/tmp/chromium+0x48d2fdd)
[pid=476][err] #34 0x5604b812bb6f (/tmp/chromium+0x48d2b6e)
[pid=476][err] #35 0x5604b812915e (/tmp/chromium+0x48d015d)
[pid=476][err] #36 0x5604b8129b1e (/tmp/chromium+0x48d0b1d)
[pid=476][err] #37 0x5604b90f88ba (/tmp/chromium+0x589f8b9)
[pid=476][err] #38 0x5604b90f85e5 (/tmp/chromium+0x589f5e4)
[pid=476][err] #39 0x7fc61725c13a __libc_start_main
[pid=476][err] #40 0x5604b68a602a _start
[pid=476][err] r8: 0000000000000010 r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000246
[pid=476][err] r12: 00007ffedd13c800 r13: 0000173400edf0b8 r14: 0000173400edf930 r15: 00007ffedd13c7d0
[pid=476][err] di: 00001734002e4000 si: 0000000000000000 bp: 00007ffedd13c840 bx: 0000173400ede800
[pid=476][err] dx: d015f40034170000 ax: 00001734002e4000 cx: 0000000000000000 sp: 00007ffedd13c770
[pid=476][err] ip: 00005604b7d6610a efl: 0000000000010246 cgf: 002b000000000033 erf: 0000000000000004
[pid=476][err] trp: 000000000000000e msk: 0000000000000000 cr2: 00000000000000b0
[pid=476][err] [end of stack trace]

[jznadams]

I had the same error when using Lambda docker image on Apple M1, you need to add the --platform linux/amd64 in order to get rid of this error 👍

@Antoine-C Where do you add that flag?

[rehanvdm]

I'm wondering if it might not be better to run puppeteer in a container now that Lambda supports it? This PR has been open and struggling for months now :( or will the same issues be present running it in a container?

PS. I am currently running this on a Lambda in production and don't want to go the container route.

[Antoine-C]

I had the same error when using Lambda docker image on Apple M1, you need to add the --platform linux/amd64 in order to get rid of this error 👍

@Antoine-C Where do you add that flag?

In your Dockerfile next to the FROM statement or as a parameter when using docker run command

[Sparticuz]

Any ideas on why the binaries from @Sparticuz's fork does this with Playwright?

I haven't used Playwright, but I also haven't changed anything that would affect it.

For people seeing slowness, does turning webgl off help? This may just be related to the new driver. If someone is running x-ray on their function, I'd love to see any info about what exactly is slow.

[yakirza17]

Hi @Sparticuz, I make tests for the page.screenshot issue and these are the numbers:

Puppeteer 14.3 with WebGL (based your repo, on tag v14.3.0, not contains the PR of decrease chrome size): avg(duration): 1775.1579 min(duration): 485 max(duration): 15519 count(duration): 3693

Old puppeteer version (5.5, without this PR): avg(duration): 275.1661 min(duration): 22 max(duration): 5097 count(duration): 3649

Puppeteer 14.3 without WebGL (--disable-webgl and remove --use-gl=angle, --use-angle=swiftshader): avg(duration): 732.9199 min(duration): 283 max(duration): 5977 count(duration): 3672

[Sparticuz]

My guess on the slowness would just be from Chrome's version creep? Those numbers are interesting @yakirza17, however, I'm looking more for a nodejs firegraph that could tell us whether the slowdown is occurring in chrome_aws_lambda, puppeteer, or chromium. FWIW, I really haven't changed any chrome_aws_lambda code besides the patch making it compatible with node16 and ts4.4+.

Edit: Puppeteer 5.5 is Chromium 88, Puppeteer 14.3 is Chromium 103. That being said, I would recommend running with webgl disabled if you don't need it.

[Antoine-C]

@Sparticuz I was able to run 0x in a Lambda docker container and the flamegraph doesn't show any unoptimized bits regarding the screenshot in playwright nor chrome_aws_lambda so IMO the slowdown is coming from chromium.

When I was using swiftshader directly --use-gl=swiftshader with chromium 99 I was already experiencing a slowness regarding the screenshots. Moving to --use-gl=angle --use-angle=swiftshader fixed it but the WebGL issue came up and it was, as you said, because swiftshader libs where not at the right place and then the slowness appeared again so I'm thinking Swiftshader with chromium 95+ might not be the right choice to use for headless (or maybe it's broken) ?

I came up across these possible flags for ANGLE : https://chromium.googlesource.com/angle/angle/+/main/doc/DebuggingTips.md, I don't have much knowledge here but do you think it's possible to use another renderer like Vulkan (given we're including some vk_* files) ? Or swiftshader is the only way to go ?

[Sparticuz]

Sorry all, I'm out of the country at the moment. I will update to the latest puppeteer next week.

[Antoine-C]

@Sparticuz No worries, have you had the chance to look at it ? On my end I've noticed quite some "Browser closed" errors with [pid=348][err] [0626/041651.533068:ERROR:egl_util.cc(74)] Failed to load GLES library: /tmp/playwright/libGLESv2.so: /tmp/playwright/libGLESv2.so: cannot open shared object file: No such file or directory but this is linked to the issue above with WebGL and files not being in the right place but causing an unexpected slowness if they are.