jmd_imagescraper icon indicating copy to clipboard operation
jmd_imagescraper copied to clipboard

Add user agent at getting search token

Open elsemieni opened this issue 11 months ago • 0 comments

I had a cron scrapping images from DuckDuckGo while suddenly stopped with the following error:

Traceback (most recent call last):
  File "/path/to/jmd_imagescraper/core.py", line 211, in duckduckgo_search
    links = duckduckgo_scrape_urls(keywords, max_results, img_size, img_type, img_layout, img_color)
  File "/path/to/jmd_imagescraper/core.py", line 82, in duckduckgo_scrape_urls
    assert match is not None, "Failed to obtain search token"
AssertionError: Failed to obtain search token

This was caused as DuckDuckGo returned a non-valid answer for https://duckduckgo.com/418.htm , which then was unable to obtain the token. I also saw other users of your library that obtained that error suddenly but they did not reported that.

This issue can be easily be solved at adding any user agent at that inital request. Here is my suggestion:

resp = requests.post(BASE_URL, data=params, headers={'user-agent': 'my-cool-user-agent/1.0.0'})

Tested that and worked again.

elsemieni avatar Mar 14 '24 01:03 elsemieni