gpt-researcher unstable api.tavily.com/search

Over the last hour I have been trying
gpt-researcher but it seems the tavily is quite unstable. it sometimes return results, but so far, I have not been able to finish a research. There is always a search that fail. Should retry be implemented? Is there is a way to restart a research that has been interrupted?

Here is an example when it fail:

  File "/Users/xx/.virtualenvs/gpt-researcher/lib/python3.11/site-packages/tavily/tavily.py", line 47, in search
    return self._search(query, search_depth=search_depth, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xx/.virtualenvs/gpt-researcher/lib/python3.11/site-packages/tavily/tavily.py", line 41, in _search
    response.raise_for_status()  # Raises a HTTPError if the HTTP request returned an unsuccessful status code
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xx/.virtualenvs/gpt-researcher/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

Dec 04 '23 14:12 jeremi

Hey @jeremi due to high volumes we had some slow response times. After scaling up it should be better now, can you check?

Dec 05 '23 13:12 assafelovic

It seems good today, no issue. Thanks!

Dec 05 '23 14:12 jeremi

Hmm looks like its recurring on my end though

🔎 Running research for 'Current market opinion on Samsung Electronics stock December 2023'... ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/uvicorn/protocols/websockets/wsproto_impl.py", line 234, in run_asgi result = await self.app(self.scope, self.receive, self.send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in call await super().call(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/applications.py", line 122, in call await self.middleware_stack(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 149, in call await self.app(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in call raise exc File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in call await self.app(scope, receive, sender) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in call raise e File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in call await self.app(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 718, in call await route.handle(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 341, in handle await self.app(scope, receive, send) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 82, in app await func(session) File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/fastapi/routing.py", line 325, in app await dependant.call(**values) File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/backend/server.py", line 50, in websocket_endpoint report = await manager.start_streaming(task, report_type, websocket) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/utils/websocket_manager.py", line 51, in start_streaming report = await run_agent(task, report_type, websocket) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/utils/websocket_manager.py", line 63, in run_agent report = await researcher.run() ^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/master/agent.py", line 52, in run scraped_sites = await self.scrape_sites_by_query(sub_query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/master/agent.py", line 91, in scrape_sites_by_query search_results = retriever.search(max_results=self.cfg.max_search_results_per_query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/c/Users/ohsam/Documents/Github/gpt-researcher/gpt_researcher/retrievers/tavily_search/tavily_search.py", line 43, in search results = self.client.search(self.query, search_depth="advanced", max_results=max_results) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/tavily/tavily.py", line 47, in search return self._search(query, search_depth=search_depth, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/tavily/tavily.py", line 41, in _search response.raise_for_status() # Raises a HTTPError if the HTTP request returned an unsuccessful status code ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/oramtristankwon/miniconda3/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

Dec 07 '23 10:12 Tristandiscorder

EchoingTristandiscorder, this seems to be persistent

Dec 17 '23 23:12 samgoodman

File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/requests/models.py", line 1021, in raise_for_status raise HTTPError(http_error_msg, response=self) requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search

I have the same problem.

Jan 19 '24 05:01 garmanwu

This is consistently happening to me as well, @assafelovic maybe worth reopening? Not sure if there is another issue open for this

Jan 30 '24 16:01 sameigen

Yep, same. @assafelovic this should be re-opened.

Not usable at current state.

Feb 24 '24 11:02 SimonB97

Can't even ping the API endpoint at the moment. Not sure what's happening with tavily. Yet it was working earlier the other day. 😓😮‍💨

It's kinda unstable since it's failure breaks the AI chain

🤦🏾‍♂️

Feb 26 '24 00:02 seanmavley

Happening to me as well. It works once and then breaks the next time. It also uses one of my API call requests when it doesn't work @assafelovic

Mar 01 '24 00:03 AryanPadmanabhan

Also getting requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.tavily.com/search. Please re-open. I get this when using both a free and paid-for API key.