Duckduckgo issue

[gleitz]

What happened:

Duckduckgo seems to be blocking requests

What you expected to happen:

Request should succeed

Output with --explain

» python -m howdoi make a pizza --engine duckduckgo --explain
INFO: Version: 2.0.16
INFO: Fetching answers for query: make a pizza
INFO: Searching duckduckgo with URL: https://duckduckgo.com/html?q=site:stackoverflow.com%20make%20a%20pizza&t=hj&ia=web
INFO: Received HTTPError
ERROR: Unable to find an answer because the search engine temporarily blocked the request. Please wait a few minutes or select a different search engine.
Traceback (most recent call last):
  File "/Users/gleitz/.homebrew/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/gleitz/.homebrew/Cellar/[email protected]/3.9.5/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/gleitz/projects/howdoi/howdoi/__main__.py", line 3, in <module>
    command_line_runner()
  File "/Users/gleitz/projects/howdoi/howdoi/howdoi.py", line 788, in command_line_runner
    utf8_result = howdoi(args).encode('utf-8', 'ignore')
  File "/Users/gleitz/projects/howdoi/howdoi/howdoi.py", line 615, in howdoi
    res = _get_answers(args)
  File "/Users/gleitz/projects/howdoi/howdoi/howdoi.py", line 418, in _get_answers
    question_links = _get_links_with_cache(args['query'])
  File "/Users/gleitz/projects/howdoi/howdoi/howdoi.py", line 397, in _get_links_with_cache
    links = _get_links(query)
  File "/Users/gleitz/projects/howdoi/howdoi/howdoi.py", line 282, in _get_links
    raise BlockError('Temporary block by search engine')
howdoi.howdoi.BlockError: Temporary block by search engine

When I open the page referenced, I occasionally get the error

If this error persists, please let us know: [email protected]

[ShezaMunir]

Reminder: once this issue is resolved, enable DDG test case in test_howdoi.py

[SaurusXI]

@gleitz perhaps using a headless browser like puppeteer or selenium might fix this problem? Not sure how it would affect performance, but might be worth looking into.

[alirazacodes]

Hi @gleitz, I would like to contribute to resolving this issue;

• I have done some research and I think if we use some data scrapping python libraries like scrappy, beautifulsoap etc
• I can also use request-html along with selenium for better data rendering
• Another method that also came along the way of my research was by using API like SerpApi

Please do assign me this issue, I have already started working on it, having some issues while running howdoi in my local machine due to some python packages being broken while installation which will be resolved by today.