[OPTIMISATION] fetch platform specific chrome image

[realityzero]

desc: Environment specific chromium image selection on development environment

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
tiny-helpers	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Mar 14, 2024 8:29am

[stefanjudis]

@realityzero Welcome to TH and thank you so much for contributing. 💯

This topic has been on the list for ages so I'm exited that someone is tackling it. I've left some comments and on the way and will test is locally once we discussed these.

Thanks again and have a great day!

[realityzero]

@realityzero Welcome to TH and thank you so much for contributing. 💯

This topic has been on the list for ages so I'm exited that someone is tackling it. I've left some comments and on the way and will test is locally once we discussed these.

Thanks again and have a great day!

woahh @stefanjudis this made my day

[stefanjudis]

@realityzero I just checked out the branch and have run vercel dev (I should document this 🤦‍♂️ ).

Unfortunately, it's not working locally and I'm running in target closed errors.

Error log

> Ready! Available at http://localhost:3000
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
TargetCloseError: Protocol error (Page.captureScreenshot): Target closed
    at CallbackRegistry.clear (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/Connection.ts:186:30)
    at CDPSessionImpl._onClosed (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/Connection.ts:611:21)
    at Connection.#onClose (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/Connection.ts:363:15)
    at WebSocket. (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/NodeWebSocketTransport.ts:60:22)
    at callListener (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:290:14)
    at WebSocket.onClose (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/node_modules/ws/lib/event-target.js:220:9)
    at WebSocket.emit (node:events:517:28)
    at WebSocket.emit (node:domain:489:12)
    at WebSocket.emitClose (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:258:10)
    at Socket.socketOnClose (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/node_modules/ws/lib/websocket.js:1264:15)
Error: Navigation failed because browser has disconnected!
    at new LifecycleWatcher (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/LifecycleWatcher.ts:109:11)
    at Frame.goto (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/Frame.ts:108:21)
    at CDPPage.goto (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/Page.ts:923:35)
    at getScreenshot (/Users/stefanjudis/Projects/github.com/tiny-helpers/api/screenshot.js:62:14)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async module.exports (/Users/stefanjudis/Project
    ...
    ...

What's your local setup to test this?

[realityzero]

@stefanjudis Apologies for getting onto this quite late. I couldn't test the solution end to end locally from tiny-helpers project. However, I followed the same abstracted approach of picking platform specific chromium in one of my recent projects: ScrapifyX.

That's my bad.

Could you please try this out: Stackoverflow

Or just tell me how to invoke screenshot.js after running this project. I was unable to find any references from UI.

[stefanjudis]

@realityzero

Apologies for getting onto this quite late.

No worries. I'm still very grateful that you're taking this on! 🎉

Or just tell me how to invoke screenshot.js after running this project.

That's my bad. As this project is deployed to vercel I run it via the CLI which I have installed globally.

$ vercel dev
Vercel CLI 33.6.0
> Running Dev Command “npx @11ty/eleventy --serve --watch --port $PORT”
Writing _site/index.html from ./site/index.html.
Writing _site/home/latest/index.html from ./site/tag-sorted-by-date.njk.
Writing _site/home/index.html from ./site/tag.njk.
...
...
[Browsersync] Access URLs:
 ----------------------------------------
       Local: http://localhost:3000
    External: http://192.168.178.88:3000
 ----------------------------------------
          UI: http://localhost:3001
 UI External: http://localhost:3001
 ----------------------------------------
[Browsersync] Serving files from: _site
> Ready! Available at http://localhost:3000
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
exePath: /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
TargetCloseError: Protocol error (Page.captureScreenshot): Target closed
    at CallbackRegistry.clear (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/common/CallbackRegistry.ts:93:30)
    at CdpCDPSession._onClosed (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/cdp/CDPSession.ts:149:21)
    at Connection.#onClose (/Users/stefanjudis/Projects/github.com/tiny-helpers/node_modules/puppeteer-core/src/cdp/Connection.ts:205:15)
...

I also looked briefly into this and no SO magic helped out for the local screenshoting. :/

[realityzero]

@stefanjudis Whenever I've read TargetCloseError or Target closed in Puppeteer, it is almost always a memory issue. Can you try adding adding following args:

1. Let's just add this first --disable-dev-shm-usage
2. If it still breaks, add one more argument --shm-size=3gb

Ref:

https://github.com/puppeteer/puppeteer/issues/1790 https://github.com/puppeteer/puppeteer/issues/1175

[stefanjudis]

@realityzero Unfortunately, neither of the options work.

I'm still running into the same exceptions. Did you try the Vercel CLI to see how to tackle it?

[realityzero]

Hey @stefanjudis ! Apologies man for such a late reply. Certain life challenges.

I did some debugging locally in two different stages.

1. Look at this diff (github wasn't allowing me to upload a .patch file)

diff --git a/api/screenshot.js b/api/screenshot.js
index c285f0c..8be5f87 100644
--- a/api/screenshot.js
+++ b/api/screenshot.js
@@ -27,7 +27,7 @@ async function getOptions(isProd) {
         options = {
           args,
           executablePath: exePath,
-          headless: 'new',
+          headless: false,
         };
     }
     return options;
@@ -57,7 +57,7 @@ function checkUrl(string) {
   return true;
 }
 
-export async function getScreenshot(url, ratio = 1) {
+async function getScreenshot(url, ratio = 1) {
   const page = await getPage();
   await page.goto(url, {
     waitUntil: 'domcontentloaded',
@@ -68,6 +68,8 @@ export async function getScreenshot(url, ratio = 1) {
     devicePixelRatio: ratio,
   });
   const file = await page.screenshot();
+  const base64Image = new Buffer(file).toString('base64');
+  console.log('done w/ SS:', base64Image);
   return file;
 }
 
@@ -92,3 +94,6 @@ module.exports = async (req, res) => {
       );
   }
 };
+
+
+getScreenshot('https://news.ycombinator.com/item?id=39894820#39916444', 1);
\ No newline at end of file

I simply executed node screenshot.js. Here I was able to get a base64 and used some base64 to image converted online and it went fine. 2. I tried vercel dev command to replicate the same connection level thing of deployment. However I couldn't navigate to a page that takes screenshots. I might need help in that. But it triggered screenshot.js behind the scenes in terminal and executed same thing as point 1.

How about we get over a call for this in the coming week?

[stefanjudis]

@realityzero Hey, so sorry for letting this slip through. :/ Life's pretty wild on my end.

But in the meantime, something changed on vercel's side and things broke entirely. I think I'll move the screenshots off serverless fns. It's just to briddle. ;/

Sorry for wasting your time.