scraper
scraper copied to clipboard
get-set-fetch
Nodejs web scraper. Contains a command line, docker container, terraform module and ansible roles for distributed cloud scraping. Supported databases: SQLite, MySQL, PostgreSQL. Supported headless cli...
current [notes](https://github.com/get-set-fetch/scraper/blob/main/development.md) are incomplete
When running against selectors like ` [ ".classA > .classB", ".classA > .classC" ] ` getSelectorBase will return an invalid selector: `"classA > "`. Don't rely (just) on space to...
right now SchemaType generating a typescript type based on a json-schema doesn't support the "required" field. both required and optional json-schema properties result in optional typescript properties. translate required json-schema...
right now all acceptance test perform scraping with the builtin plugins. add new test defining a custom plugin with a node_modules dependency in order to test plugin bundling.
as plugins and plugin options change, versioning will warn about possibly no longer valid scrape definitions
add a ClickPlugin responsible for performing javascript click actions
Metadata
Owner
Metadata
Nodejs web scraper. Contains a command line, docker container, terraform module and ansible roles for distributed cloud scraping. Supported databases: SQLite, MySQL, PostgreSQL. Supported headless cli...