scrapix

scrapix copied to clipboard

Is the `settings_sent` parameter of the schema strategy used?

1

In the schema strategy there is an option named `settings_sent` that does not appear to be used in the code. Is it still somehow used?

bidoubiwa

Create multiple handlers for different types of page content. Ask the user which template to use. If there is no template, use the content page default one. - [x] #6...

qdequele

Where is `only_type` located in the schema settings option

1

Previously the `only_type` was fetched in `this.config.schema_config?.only_type`, nonetheless, using typescript it appears the config is located in `this.config.schema?.only_type`. This should be tested. ```js if (this.config.schema?.only_type) { // TODO: Might be...

bidoubiwa

Does the `exctract_code_from_page` feature works?

1

In this PR #28, I had to fix the call made to `this._format_code(code)` that was wrong. It previously was `format_code(code);`. Which does not exists. This leads me to think that...

bidoubiwa

url_to_index does not work

1

When adding `urls_to_index` to my configuration file, no pages are indexed at all

bidoubiwa

Retrieve page titles from meta tags

3

in the results, the page hosted at `website.com/docs/examples/foo` will be formatted (written in pink) as `docs / examples / foo` (or `docs > examples > foo`, depending on which formatting...

Strift

Currently if the redis server is not answering it is silently ignored and nothing happens. Which makes debugging harder.

bidoubiwa

When a page is not found during build, the page is replaced with a 404 displayed theme. When accessing this page, the HTTP code is not 404 but 200. Because...

bidoubiwa

## Context In the current state of this project, the scraping works and does its job as intended. Nonetheless, some essential parts are missing for this repository to become a...

bidoubiwa

Add route to empty the task queue

1

Sometimes it might be useful to cancel the tasks queue. It happened when I send the wrong meilisearch credentials but the processes were already launched. This is the way to...

bidoubiwa