scrapy-puppeteer
scrapy-puppeteer copied to clipboard
ispras
Library that helps use puppeteer in scrapy.
# Description As we fixed CustomJsAction response in the JavaScript part, we do not need search through the all attributes of Response. This will increase performance and readability. ## Type...
Hi, is there a way i can custom the puppeteer request, for example to add puppeteer extra functionality? thank you.
# Description RecaptchaMiddleware has a small bug. It has ignored close_page=True attribute of some PuppeteerRequests. Also beautified process_request/process_response function signatures of middlewares. ## Type of change - [x] Bug fix...
# Description Implemented new downloader middleware for restoring contexts when they are destroyed. Added example spider. Added info about the middleware to README. Restructured the project to have more than...
# Description Implemented `Compose` action which is capable of combining several PuppeteerActions into one. Also I have fixed several bugs. ## Type of change - [x] Bug fix (non-breaking change...
# Description In this PR I have improved `follow` method of `PuppeteerResponse` class. Also I have added `follow_all` method in it. Updated documentation. ## Type of change - [x] New...
# Description Added wait_options and navigation_options to recaptcha_solver action. Added them in RecaptchaSolverMiddleware. A bit of documentation is updated. ## Type of change Please delete options that are not relevant....
# Description Refactored browser managers so they now do not use `syncer` and `bs4`. They do not skip middlewares after PuppeteerServiceDownloaderMiddleware. They do not log useless messages to the console....