SSL error for Semantic Scholar

[Shellfishgene]

Hi!

I'm using the command line tool, like pqa index or ask and always get an SSL error for semantic scholar: ClientConnectorCertificateError: Cannot connect to host api.semanticscholar.org:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1002)')] Is this problem on my side, with semantic scholar, or paper-qa? Or do I need an API key?

Cheers

[dosubot[bot]]

Hey @Shellfishgene! 👋 I'm here to help you out with bugs, questions, and becoming a contributor. Let's squash those bugs together!

The SSL error you're encountering is likely related to your setup or the Semantic Scholar API. The paper-qa tool does not have specific SSL configurations, so the issue is probably due to the environment in which the tools are being run.

Here are some steps to resolve the SSL error:

1. Update SSL Certificates: Ensure that your system's SSL certificates are up to date.
2. Network Configuration: Verify that your network allows SSL connections to the Semantic Scholar API.
3. Proxy Settings: If you are behind a proxy, ensure it is correctly configured to handle SSL traffic.

While an API key is optional, it is recommended to avoid rate limits. You can set the SEMANTIC_SCHOLAR_API_KEY environment variable to use an API key:

export SEMANTIC_SCHOLAR_API_KEY=your_semantic_scholar_api_key

However, the SSL error is not directly related to the presence or absence of an API key ^[1][2].

To continue talking to Dosu, mention @dosu.

[jamesbraza]

@Shellfishgene I am not sure. To proceed, can you answer:

• Do you have the SEMANTIC_SCHOLAR_API_KEY environment variable set?
• Do you get the same outcome on different internet connections?

[DGoettlich]

hi sorry to warm this up again – I am getting the same error; API Key is set and i get the same error on different internet connections. Should add that I didn't have the problem a few weeks ago.

[jamesbraza]

Can you share your full stack? And is the error flaky, or does it always happen?

[DGoettlich]

paper-qa = 5.0.5, python = 3.12, running on MacOS in a poetry venv – let me know if you need more details.

I used paper-qa intensively last month and didn't have the problem. Right now, the error is persistent. I'm getting this message (slightly abbreviated).

Retrying paperqa.clients.semantic_scholar._s2_get_with_retrying in 0.0 seconds as it raised ClientConnectorCertificateError: Cannot connect to host api.semanticscholar.org:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')]. [...].

Traceback (most recent call last): File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/aiohttp/connector.py", line 1098, in _wrap_create_connection return await self._loop.create_connection(*args, **kwargs, sock=sock) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 1146, in create_connection transport, protocol = await self._create_connection_transport( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 1179, in _create_connection_transport await waiter File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/sslproto.py", line 575, in _on_handshake_complete raise handshake_exc File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/sslproto.py", line 557, in _do_handshake self._sslobj.do_handshake() File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ssl.py", line 917, in do_handshake self._sslobj.do_handshake() ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 114, in call result = await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/clients/semantic_scholar.py", line 119, in _s2_get_with_retrying return await _get_with_retrying( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 189, in async_wrapped return await copy(fn, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 111, in call do = await self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 153, in iter result = await action(retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/_utils.py", line 99, in inner return call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/init.py", line 398, in self._add_action_func(lambda rs: rs.outcome.result()) ^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 114, in call result = await fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/utils.py", line 450, in _get_with_retrying async with session.get(url, **get_kwargs) as response: File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/aiohttp/client.py", line 1359, in aenter self._resp: _RetType = await self._coro ^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/aiohttp/client.py", line 663, in _request conn = await self._connector.connect( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/aiohttp/connector.py", line 563, in connect proto = await self._create_connection(req, traces, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/aiohttp/connector.py", line 1032, in _create_connection _, proto = await self._create_direct_connection(req, traces, timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/aiohttp/connector.py", line 1366, in _create_direct_connection raise last_exc File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/aiohttp/connector.py", line 1335, in _create_direct_connection transp, proto = await self._wrap_create_connection( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/aiohttp/connector.py", line 1100, in _wrap_create_connection raise ClientConnectorCertificateError(req.connection_key, exc) from exc aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host api.semanticscholar.org:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)')]

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Users/daniel/Documents/Research Projects/RAG/rag_pipeline/code/init_rag.py", line 102, in docs.add(doc_path, docname=doc_name, settings=settings) File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/docs.py", line 227, in add return get_loop().run_until_complete( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 684, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/docs.py", line 364, in aadd doc = await metadata_client.upgrade_doc_to_doc_details( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/clients/init.py", line 210, in upgrade_doc_to_doc_details if doc_details := await self.query(**kwargs): ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/clients/init.py", line 153, in query await gather_with_concurrency( File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/utils.py", line 114, in gather_with_concurrency return await asyncio.gather(*(sem_coro(c) for c in coros)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/utils.py", line 112, in sem_coro return await coro ^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/clients/client_models.py", line 109, in query return await self._query(client_query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/clients/semantic_scholar.py", line 350, in _query return await get_s2_doc_details_from_title( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/clients/semantic_scholar.py", line 335, in get_s2_doc_details_from_title return await s2_title_search( ^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/paperqa/clients/semantic_scholar.py", line 238, in s2_title_search data = await _s2_get_with_retrying( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 189, in async_wrapped return await copy(fn, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 111, in call do = await self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 153, in iter result = await action(retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/_utils.py", line 99, in inner return call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/daniel/Library/Caches/pypoetry/virtualenvs/rag-pipeline-bPftndy5-py3.12/lib/python3.12/site-packages/tenacity/init.py", line 419, in exc_check raise retry_exc from fut.exception() tenacity.RetryError: RetryError[<Future at 0x124c07e30 state=finished raised ClientConnectorCertificateError>]

[DGoettlich]

turns out the simple fix is /Applications/Python\ 3.12/Install\ Certificates.command