firecrawl icon indicating copy to clipboard operation
firecrawl copied to clipboard

[Feat] Remote Playwright Instances

Open mschfh opened this issue 4 months ago • 1 comments

Problem Description Firecrawl lacks support for connecting to remote playwright instances via CDP.

Proposed Feature Implement support for remote Playwright connections over CDP (Chrome DevTools Protocol) to external browser instances.

Implementation Suggestions

  • add an env var to configure a CDP endpoint
  • use the CDP endpoint instead of launching a local browser

Use Case

  • Use Bright Data scraping browser or Zenrows to circumvent captchas, bot detection and other restrictions.

Additional Context

  • Bright Data's scraping browser: https://docs.brightdata.com/scraping-automation/scraping-browser/code-examples
  • Zenrows scraping browser: https://www.zenrows.com/solutions/scraping-browser
  • Playwright's connect_over_cdp: https://playwright.dev/python/docs/api/class-browsertype#browser-type-connect-over-cdp

mschfh avatar Oct 04 '24 23:10 mschfh