website-scraper-puppeteer

website-scraper-puppeteer copied to clipboard

Bumps [puppeteer](https://github.com/puppeteer/puppeteer) from 15.5.0 to 16.1.0. Release notes Sourced from puppeteer's releases. v16.1.0 16.1.0 (2022-08-06) Features use an xpath query handler (#8730) (5cf9b4d) Bug Fixes resolve target manager init if...

dependabot[bot]

Page Timeout

1

Hi Sophie, Is there a way to set timeout when accessing the page (i.e. at `page.goto(url)`) without modifying it manually in the API.

Germminate

Some websites, eg: google.com don't get scraped if i pass custom useragent headers ` request: { headers: { 'User-Agent': "Mozilla/5.0 (Macintosh; Intel Mac OS X 11_15) AppleWebKit/537.36 (KHTML, like Gecko)...

shreejalmaharjan-27

Suggestion: catch dynamic load contents

4

While puppeteer is used inside this plugin, it should be able to monitor files that loaded by javascript, and add it to the download list, not only the ones written...

zhangciwu

Whats the best approach to speed things up?

2

Hi, I'm looking for ways to speedup the crawling process. Where website-scraper takes up to 8 minutes to crawl a site, website-scraper-puppeteer needs 40 minutes for the same site. (sure...

jalbstmeijer

Problems of Redirection

1

There is a "page turn button" that implemented with javascript, and it redirects to another page on the website. The problem is that the script saves it to the "redirection...

LZY2006

[Investigation] Fix failing tests by timeout

1

aivus

Bumps [puppeteer](https://github.com/puppeteer/puppeteer) from 16.2.0 to 17.1.3. Release notes Sourced from puppeteer's releases. v17.1.3 17.1.3 (2022-09-08) Bug Fixes FirefoxLauncher should not use BrowserFetcher in puppeteer-core (#8920) (f2e8de7), closes #8919 linux arm64...

dependabot[bot]

Bumps [puppeteer](https://github.com/puppeteer/puppeteer) from 21.11.0 to 22.2.0. Release notes Sourced from puppeteer's releases. puppeteer-core: v22.2.0 22.2.0 (2024-02-21) Features roll to Chrome 122.0.6261.57 (r1250580) (#11958) (70ad3b2) Bug Fixes deprecate isIncognito (#11962) (ceab7a9)...

dependabot[bot]

Bumps [chai](https://github.com/chaijs/chai) from 4.4.1 to 5.1.0. Release notes Sourced from chai's releases. v5.1.0 What's Changed Remove useless guards and add parentheses to constuctors by @​koddsson in chaijs/chai#1593 Cleanup jsdoc comments...

dependabot[bot]