scrapix icon indicating copy to clipboard operation
scrapix copied to clipboard

url_to_index does not work

Open bidoubiwa opened this issue 2 years ago • 1 comments

When adding urls_to_index to my configuration file, no pages are indexed at all

bidoubiwa avatar Jun 28 '23 18:06 bidoubiwa

I can't reproduce.

When I try this config the following config, I have 950 docs:

{
    "start_urls": [
        "https://meilisearch.com/docs",
        "https://www.meilisearch.com/docs"
    ],
    "meilisearch_url": "{{meilisearch_host}}",
    "meilisearch_api_key": "{{meilisearch_api_key}}",
    "meilisearch_index_uid": "{{meilisearch_index_name}}"
}

But when I try this config the following config, I have 468 docs:

{
    "start_urls": [
        "https://meilisearch.com/docs",
        "https://www.meilisearch.com/docs"
    ],
    "urls_to_index": ["https://www.meilisearch.com/docs/learn"],
    "meilisearch_url": "{{meilisearch_host}}",
    "meilisearch_api_key": "{{meilisearch_api_key}}",
    "meilisearch_index_uid": "{{meilisearch_index_name}}"
}

qdequele avatar Jul 22 '23 12:07 qdequele