dude

dude copied to clipboard

Output is always flattened into a single list of dictionaries before saving to CSV, JSON, etc. By grouping data into separate tables, it will be easier to post-process and merge...

roniemartinez

It will be easier to extract individual table cell data if `colspan` and `rowspan` are exploded into single cells. Add options (`--explode-rowspan`, `--explode-colspan` and/or `--explode-table-cells`) to use. For example, this...

roniemartinez

✨ Match or skip handler (function) names

1

Resolves #29

roniemartinez

A good reference: https://github.com/TeamHG-Memex/MaybeDont/blob/master/maybedont/predict.py

roniemartinez

There are existing ways to extract data from JSON without traversing the contents one by one. ## Options - [JsonPath](https://goessner.net/articles/JsonPath/) - [JMESPath](https://jmespath.org/) ## Proposed style ```python @select(jsonpath="$.store.book[0].title") def extract_title(title): return...

roniemartinez

An option to run only or skip specific handler functions ```bash dude scrape ... --run-only dude scrape ... --skip ``` Ideas: - By parameter to `@select()`. - By function name

roniemartinez

https://curl.se/docs/manpage.html#URL

roniemartinez

Selenium proxy is not yet implemented https://github.com/roniemartinez/dude/blob/169181386063a90d83d3b0b985f92a2c47a1d28c/dude/optional/selenium_scraper.py#L142 https://github.com/roniemartinez/dude/blob/169181386063a90d83d3b0b985f92a2c47a1d28c/dude/optional/selenium_scraper.py#L184

roniemartinez

roniemartinez

https://github.com/AtuboDad/playwright_stealth

roniemartinez