dude

dude copied to clipboard

roniemartinez

roniemartinez

- Playwright - https://playwright.dev/docs/auth#reuse-authentication-state - Check possibility if this can be done with other backends

roniemartinez

Modified Selenium that circumvents anti-bot. Repo: https://github.com/ultrafunkamsterdam/undetected-chromedriver > Optimized Selenium Chromedriver patch which does not trigger anti-bot services like Distill Network / Imperva / DataDome / Botprotect.io Automatically downloads the...

roniemartinez

1. Download by file extension 2. Download by mimetype, e.g. `png` should also match `image/png` mimetype ```console dude scrape ... --download png,jpg # download all png and jpg files dude...

roniemartinez

[Autoscraper](https://github.com/alirezamika/autoscraper) is made for automatic web scraping to make scraping easy. I believe it would be incredible to also include it.

daniel7an

Study: ML-based scrapers

6

## Possible format: ```python @select(sample="path/to/training/data") def handler(result): return {"data": result} ``` ## Potential backends: - https://github.com/lorey/mlscraper

roniemartinez

https://www.reddit.com/r/Python/comments/tc3x72/comment/i0xvy98/?utm_source=share&utm_medium=web2x&context=3

roniemartinez

https://github.com/browserless/chrome#hosting-providers

roniemartinez

User-Agent

7

- Set a value for Dude User-Agent instead of using the default values on each parser backend (e.g.: `pydude/{version} (+https://github.com/roniemartinez/dude)`) - Add option to override the User-Agent - For Playwright...

roniemartinez