Thoughts on improved Typescript types

[UrielCh]

Hi after a year away from this project, I'm back. I'm happy that my previous PRs (#455 #461) had help in #555.

My next change will convert more remaining JS files to TS.

To avoid applying many changes, I would like to know the official way to configure puppeteer-extra-plugin-stealth's evasions options?

For example, 'stealth/evasions/navigator.hardwareConcurrency' accept a hardwareConcurrency options, but I did not find any doc about providing such a value.

[UrielCh]

Your project can not be built on a Windows host Can I fix that too?

[UrielCh]

I will drop TsLint, ESLint should be used instead.

[UrielCh]

Full change list is available in #649

[berstend]

Hey, thanks for the PR - it's indeed massive 😄 I need a moment to go through the changes, but it's definitely great as reference and potential backlog.

We have a larger change on the horizon for the stealth evasions: We need to collect their scripts first and only inject them once to help with a set of issues, so some larger changes will happen here which I need to clarify first.

[UrielCh]

Hi, happy to see you back.

Maybe this time you will end by clicking the merge button.

I also rename my jQuery plugin as puppeteer-extra-plugin-jquery so it can be loaded using puppeteer-extra.

[UrielCh]

Last update no more regression, older plugins are now compatible.

[berstend]

Lots of good stuff in there 👍 Going through it as I find the time

[UrielCh]

Tell me what do you need for a quick merge, I need to use my fork quickly. Or I Will need to add a support custom plugin naming to use a forked version.

I see that you start some refreshing on the repo.

Can you tell me why are you keeping puppeteer-firefox ? the Project is totaly dead.

I try to work on my Mac M1 but arm current chromium is crashing all the time.

I guess that you are working on an X86 Mac. Can you at least try to make it works on M1/M2 Mac ?

[UrielCh]

I see that you add some: "types": "index.d.ts", VS my "typings": "index.d.ts",

I can read here

Note that the "typings" field is synonymous with types, and could be used as well.

Also note that if your main declaration file is named index.d.ts and lives at the root of the package (next to index.js) you do not need to mark the types property, though it is advisable to do so.

older package.json contains: "typings": "dist/index.d.ts", not "types": "dist/index.d.ts",

please do not generate more conflict than needed.

[UrielCh]

you gave me extra work with your pretty useful last changes :wink:

can you explain me this change: const client = page && typeof page._client === 'function' ? page._client() : page._client

Which version of pptr has _client as a function and which one as a field? I suppose that page._client() is for pptr14+

[berstend]

Since [email protected]: page._client changed to page._client()

[berstend]

Tell me what do you need for a quick merge, I need to use my fork quickly. Or I Will need to add a support custom plugin naming to use a forked version.

I don't think I can merge that quickly - what's the most pressing thing you need that makes you consider forking?

Can you tell me why are you keeping puppeteer-firefox ?

I'm effectively the sole maintainer and there's usually more pressing things than cleaning up old stuff like that :) Is it breaking anything for you or just a nuisance seeing it still being mentioned?

I try to work on my Mac M1 but arm current chromium is crashing all the time.

My main machine is an M1 but arm browser binaries are only supported in later puppeteer/playwright releases, hence I use rosetta (just duplicate your iterm2/terminal app and make that copy open with rosetta) - that works pretty well

I see that you start some refreshing on the repo.

I'm kicking back into high gear yeah :D I'm effectively solo-maintaining this project so when I'm busy with other things that means the project progress is pausing more less.

I would like to understand what your most pressing concerns are you're addressing with your PR?

I've worked on that massive automation-extra branch & project rewrite last year but it became outdated fairly quickly as playwright & puppeteer are progressing (and in a sense diverging) fast.

A consistent problem has been typescript & typing related: We internally need to use puppeteer/playwright types yet we don't want to ship with hardcoded ones as the end-user should be able to use whatever driver version they want. I think I found a solid solution here and currently work on other similar issues that caused problems in the past, to be used as future building blocks.

Overall I'm looking into more incremental changes rather than massive PRs (that applies to the ones I created as well lol)

I'd be happy to discuss more on what I have in mind for the future and how your code might tie into that

[UrielCh]

I finally, link my project to my puppeteer-extra using a pnpm install ../puppeteer-extra-ts/packages/.... and it works. so no more emergency for my current project.

I may get the project fakebrowser back to life, but for that, I need to merge this PR so I can work properly.

For me puppeteer-firefox is a pain, since I work on windows, there is no way I can build puppeteer-extra with puppeteer-firefox

I stop using Rosetta on my M1, all my tools are native arm, And I do not use my M1 mac for this project for now.

As you may notice on my Github account I do not write any code in Javascript, I work with Java for 10 years, so I get used to strongly typed languages. I need I convert all the projects I use to typescript before doing any feature addon.

I have my own automation API, but my API is... 16 years old, I have many projects still using it, and now I really want to kill it. I have a tremendous number of old projects to maintain.

My new automation framework will be based on a puppeteer or playwright; Migrating a decade of projects is a tedious task, and during the migration I get some new project inquiry queueing.

PS1: I have updated my GitHub action, now I test my fork on Windows / Ubuntu and MacOS, and all of them pass. PS2: I have checked your playwright branch, and adding it back to my fork is not so difficult.

[UrielCh]

The test-ci depend on https://localtunnel.me and localtunnel.me is down

[UrielCh]

Fake browser is now fully converted to typescript. But required my puppeteer-extra fork to work.

[berstend]

My new automation framework will be based on a puppeteer or playwright

Take a look at playwright-extra 😄

I've added a couple new ways to provide plugin dependencies, the regular dependencies accessor supports Arrays, Sets and Maps (so default opts can be provided), as well as a new plugins accessor that already provides plugin instances which greatly simplifies the plugin dependency logic: https://github.com/berstend/puppeteer-extra/blob/441f14cd5945c359a85914715005dd8ef7e00625/packages/playwright-extra/src/plugins.ts#L294-L309

Dynamic requires will go the way of the dodo, but they're currently still in use so I added a few niceties in the meantime

This allows overwriting the defaults for dependencies (you implemented something similar in your PR): https://github.com/berstend/puppeteer-extra/blob/441f14cd5945c359a85914715005dd8ef7e00625/packages/playwright-extra/src/plugins.ts#L92-L104

And this allows providing a plugin class for any dependency (which will then skip the dynamic require) - this is especially useful for bundlers like webpack, nexe, etc that choke on dynamic requires: https://github.com/berstend/puppeteer-extra/blob/441f14cd5945c359a85914715005dd8ef7e00625/packages/playwright-extra/src/plugins.ts#L106-L121

I'm unsure yet how to best support strong typing in something like plugins.setDependencyDefaults without shipping plugin type info in extra itself (which is what you did). It'd probably be much simpler if we used a static id property in plugins instead of an instance member but it's a bit too late to change that 😄

[berstend]

I renamed this issue so its not confused with something from me

[UrielCh]

okey, keep going.

[UrielCh]

After weeks, testing / forking lots of nodejs automation API.

I tried chromy, puppeteer, playwright, but none of those works properly on android chrome.

now I'm trying with Chrome DevTools Protocol directly. but

• chrome-remote-interface
• devtools-protocol have too mush typing error to be used.

So I start my own chrome-remote-interface (this morning), and that should be a good start.

Then I will add some pptr like method, and then add puppeteer-extra-plugin-stealth to it.

Would you be interested in such a project?

[berstend]

@UrielCh that pesky firefox dependency is gone now (#673) 😄

In regards of a custom CDP lib: I've been thinking about it for years, but besides some prototyping never pursued this further.

The issue: The existing frameworks (namely puppeteer, playwright) have huge mind share and implemented countless convenience functions (screeenshot, goto). Replicating all these (or at least the important ones) with proper documentation and tests is quite a big undertaking for little (?) gain.

I guess the custom lib would need a very strong value proposition and bring something unique to the table to be worthwhile developing (and more importantly) maintaining. :)

[UrielCh]

WIP, for now. With this approach, it's possible to connect to any existing / running automation API and inject stealth.