[New RM] Support DuckDuckGoSearchAPI and TavilySearchAPI as Alternatives to You.com

[songkq]

1. Support DuckDuckGoSearchAPI and TavilySearchAPI as Alternatives to You.com.
2. When enabling TopicExpert to use DuckDuckGoSearchAPI or TavilySearchAPI, these APIs will return compelete contents instead of snippets as default.
3. One can setup the search API through editing these environments in secrets.toml: Set WEB_SEARCH_API as one of ['DuckDuckGoSearchAPI', 'TavilySearchAPI', 'YouSearchAPI'], using YouSearchAPI as default Setup You.com search API key by YDC_API_KEY=<your_youcom_api_key> Setup api.tavily.com search API key by TAVILY_API_KEY=<your_api_tavily_com_key>

[shaoyijia]

Thanks for the effort! We will test this on our side first.

Also, we plan to have a major update to this open-source repo to make it easier to customize or build on top of STORM in the upcoming weeks. We may back up the current branch and directly merge different source support there. Will keep you posted in this PR thread.

[leonstd2015]

Thanks for your effort！ After running your version, I finnally got the result, but there still have some errors, the openai refuse connection after a while," raise error.APIConnectionError( openai.error.APIConnectionError: Error communicating with OpenAI: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))"

Have you ever consider change the model to llama3 run by ollama?

[minsixhao]

@shaoyijia Because I don't have a credit card, I can't use YOU.com. Are there any plans to integrate other search APIs in the near future?

[shaoyijia]

Hi @minsixhao, currently we also support Bing Search (see the code in rm.py). We provide scripts in examples to demonstrate how to use it.

In near future (1-2 weeks), we plan to release the support of integrating retrieval models on user provided corpus as an alternative to using search engine to retrieve information from the whole Internet. If we have time, we may test to see whether it's possible to integrate the two search engines mentioned in this PR.

[naseerfaheem]

@shaoyijia any plans to merge this? DuckDuckGo and Tavilly are great additions.

[shaoyijia]

Hi @songkq , are you willing to modify this PR so I can merge the support for TavilySearchAPI and DuckDuckGoSearchAPI? I think (1) moving DuckDuckGoSearchAPI and TavilySearchAPI to knowledge_storm/rm.py and (2) having an integrated testing to see whether the results with these two APIs look good will be enough.

Thank you so much for your help!

[Yucheng-Jiang]

Closing this PR as Tavily and DuckDuckGo are supported by #145 and #149.