search_paper takes forever and does not return any output

[MMesgar]

Hi,

Any hints on why this leads to an error?

[danielnsilva]

It's possible that the API is returning an HTTP 429 status code, which may occur when the rate limit is reached or the API service is overloaded. For unauthenticated users, the rate limit is lower.

By default, the lib will attempt to retry the request up to 10 times with a 30-second interval whenever it receives an HTTP 429 response.

If you want to change this behavior, you can set retry=False. In this case, it will only make one attempt and if it receives an HTTP 429 response, it will raise a ConnectionRefusedError exception.

from semanticscholar import SemanticScholar
sch = SemanticScholar(retry=False)
results = sch.search_paper('software engineering', limit=1)
print(len(results))

Traceback (most recent call last):
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/tenacity/_asyncio.py", line 50, in __call__
    result = await fn(*args, **kwargs)
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/semanticscholar/ApiRequester.py", line 154, in _get_data_async
    raise ConnectionRefusedError('HTTP status 429 Too Many Requests.')
ConnectionRefusedError: HTTP status 429 Too Many Requests.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/main.py", line 3, in <module>
    results = sch.search_paper('software engineering', limit=1)
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/semanticscholar/SemanticScholar.py", line 335, in search_paper
    results = loop.run_until_complete(
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
  File "/nix/store/8w6mm5q1n7i7cs1933im5vkbgvjlglfn-python3-3.10.13/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/nix/store/8w6mm5q1n7i7cs1933im5vkbgvjlglfn-python3-3.10.13/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/semanticscholar/AsyncSemanticScholar.py", line 471, in search_paper
    results = await PaginatedResults.create(
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/semanticscholar/PaginatedResults.py", line 58, in create
    await obj._async_get_next_page()
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/semanticscholar/PaginatedResults.py", line 125, in _async_get_next_page
    results = await self._request_data()
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/semanticscholar/PaginatedResults.py", line 116, in _request_data
    return await self._requester.get_data_async(
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/semanticscholar/ApiRequester.py", line 112, in get_data_async
    return await self._get_data_async.retry_with(
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/tenacity/_asyncio.py", line 88, in async_wrapped
    return await fn(*args, **kwargs)
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/tenacity/_asyncio.py", line 47, in __call__
    do = self.iter(retry_state=retry_state)
  File "/home/runner/.pythonlibs/lib/python3.10/site-packages/tenacity/__init__.py", line 326, in iter
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x766d7fe4bbb0 state=finished raised ConnectionRefusedError>]

[danielnsilva]

I will close this issue. Please re-open it if you have any questions on this issue.