se-scraper icon indicating copy to clipboard operation
se-scraper copied to clipboard

Duckduckgo wait_for_results selector doesn't work in case of no results

Open PinkSheep opened this issue 5 years ago • 1 comments

Hello It seems like the selector you are currently using to scrape the Duckduckgo Result Page is not properly working in the case that the search doesn't produce any results:

[i] duckduckgo scrapes keyword ... on page 1
Problem with scraping ... in search engine duckduckgo: TimeoutError: waiting for selector ".result__body" failed: timeout 10000ms exceeded

I think the issue here is the selector you are using:

async wait_for_results() {
        await this.page.waitForSelector('.result__body', { timeout: this.STANDARD_TIMEOUT });
    }

If the search produces no results, the Duckduckgo Results Page doesn't have a div with the class .result__body. As a fix I suggest using one of the following selectors instead (I have not tested it though, but they are present in both cases):

#links
.serp__results

PinkSheep avatar Nov 29 '19 12:11 PinkSheep

have you got a solution for this?

sasokyleader avatar Jul 17 '20 10:07 sasokyleader