feat: crawler extension

Open load1n9 opened this issue 1 year ago • 4 comments

Sep 30 '24 03:09 load1n9

:eyes: :eyes: :eyes:

I'm skimming this currently but given that scraping is my primary use for Astral I'm definitely curious.

Tell me more?

Oct 07 '24 22:10 chaosharmonic

@load1n9 Could you run deno fmt?

Oct 08 '24 15:10 lino-levan

It would probably be better here to integrate with https://crawlee.dev/ instead of reinventing the wheel.

Dec 16 '24 05:12 nnmrts

Some of this also could be deeper and cover other issues. User agent has since been covered separately, but there also looks to be what would be an implementation here of Page.loadScriptTag (potentially similar to #96), and various other stuff.

Notably, without a deeper look at this, sessionManagement seems like it could encounter the same problems laid out in #86

Dec 17 '24 03:12 chaosharmonic