🐛 Bug Report: Second conversation ssl handshake error

[yhyu13]

📜 Description

Ignore the GPT2 token exception, it is just the ssl handshake error

docsgpt-backend-1   | [2023-10-07 16:34:24 +0000] [8] [ERROR] Socket error processing request.
docsgpt-backend-1   | Traceback (most recent call last):
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 714, in urlopen
docsgpt-backend-1   |     httplib_response = self._make_request(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 403, in _make_request
docsgpt-backend-1   |     self._validate_conn(conn)
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1053, in _validate_conn
docsgpt-backend-1   |     conn.connect()
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/urllib3/connection.py", line 419, in connect
docsgpt-backend-1   |     self.sock = ssl_wrap_socket(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
docsgpt-backend-1   |     ssl_sock = _ssl_wrap_socket_impl(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
docsgpt-backend-1   |     return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/ssl.py", line 513, in wrap_socket
docsgpt-backend-1   |     return self.sslsocket_class._create(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/ssl.py", line 1104, in _create
docsgpt-backend-1   |     self.do_handshake()
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/ssl.py", line 1375, in do_handshake
docsgpt-backend-1   |     self._sslobj.do_handshake()
docsgpt-backend-1   | ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1007)
docsgpt-backend-1   |
docsgpt-backend-1   | During handling of the above exception, another exception occurred:
docsgpt-backend-1   |
docsgpt-backend-1   | Traceback (most recent call last):
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
docsgpt-backend-1   |     resp = conn.urlopen(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 798, in urlopen
docsgpt-backend-1   |     retries = retries.increment(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/urllib3/util/retry.py", line 592, in increment
docsgpt-backend-1   |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
docsgpt-backend-1   | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /gpt2/resolve/main/tokenizer_config.json (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))
docsgpt-backend-1   |
docsgpt-backend-1   | During handling of the above exception, another exception occurred:
docsgpt-backend-1   |
docsgpt-backend-1   | Traceback (most recent call last):
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 136, in handle
docsgpt-backend-1   |     self.handle_request(listener, req, client, addr)
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 194, in handle_request
docsgpt-backend-1   |     util.reraise(*sys.exc_info())
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/gunicorn/util.py", line 626, in reraise
docsgpt-backend-1   |     raise value
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/gunicorn/workers/sync.py", line 184, in handle_request
docsgpt-backend-1   |     for item in respiter:
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/werkzeug/wsgi.py", line 256, in __next__
docsgpt-backend-1   |     return self._next()
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/werkzeug/wrappers/response.py", line 32, in _iter_encoded
docsgpt-backend-1   |     for item in iterable:
docsgpt-backend-1   |   File "/app/application/api/answer/routes.py", line 143, in complete_stream
docsgpt-backend-1   |     tokens_batch = count_tokens(i["prompt"]) + count_tokens(i["response"])
docsgpt-backend-1   |   File "/app/application/api/answer/routes.py", line 62, in count_tokens
docsgpt-backend-1   |     tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1748, in from_pretrained
docsgpt-backend-1   |     resolved_config_file = cached_file(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/transformers/utils/hub.py", line 417, in cached_file
docsgpt-backend-1   |     resolved_file = hf_hub_download(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
docsgpt-backend-1   |     return fn(*args, **kwargs)
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1195, in hf_hub_download
docsgpt-backend-1   |     metadata = get_hf_file_metadata(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
docsgpt-backend-1   |     return fn(*args, **kwargs)
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1532, in get_hf_file_metadata
docsgpt-backend-1   |     r = _request_wrapper(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 407, in _request_wrapper
docsgpt-backend-1   |     response = _request_wrapper(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 442, in _request_wrapper
docsgpt-backend-1   |     return http_backoff(
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 212, in http_backoff
docsgpt-backend-1   |     response = session.request(method=method, url=url, **kwargs)
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
docsgpt-backend-1   |     resp = self.send(prep, **send_kwargs)
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
docsgpt-backend-1   |     r = adapter.send(request, **kwargs)
docsgpt-backend-1   |   File "/usr/local/lib/python3.10/site-packages/requests/adapters.py", line 517, in send
docsgpt-backend-1   |     raise SSLError(e, request=request)
docsgpt-backend-1   | requests.exceptions.SSLError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /gpt2/resolve/main/tokenizer_config.json (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1007)')))

👟 Reproduction steps

1, docker-compose build && docker-compose up with .env

VITE_API_STREAMING=true
LLM_NAME=openai

OPENAI_API_BASE=<secret>
API_KEY=<secret>

Note, I am using a redirect base url that is not offical openai. I've tried both redirection that is true openai backend and local llm backend.

2, In both case, I can do embedding for my doc as fine. When I started in the conversation box named "new chat" the first conversation prompt 'summerize doc' is fine, too.

But when it comes to the second round of conversation, DocsGPT automatically change my convseration name to what it think is fine

This looks suspisous, I am not sure if that is the reason that cause some error, that leads to later convserations fail with ssl and handshake.

👍 Expected behavior

My convseration should continue as fine, since I don't have any ssl handshake problem in the first conversation

👎 Actual Behavior with Screenshots

Posted above

💻 Operating system

Windows

What browsers are you seeing the problem on?

Microsoft Edge

🤖 What development environment are you experiencing this bug on?

Docker

🔒 Did you set the correct environment variables in the right path? List the environment variable names (not values please!)

Posted above

📃 Provide any additional context for the Bug.

No response

📖 Relevant log output

No response

👀 Have you spent some time to check if this bug has been raised before?

• [X] I checked and didn't find similar issue

🔗 Are you willing to submit PR?

None

🧑‍⚖️ Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[parag477]

Can I take up this?

[dartpain]

@yhyu13 I think its because of

Note, I am using a redirect base url that is not offical openai. I've tried both redirection that is true openai backend and local llm backend.

Currently OPENAI_API_BASE url is only for azure endpoints. Maybe we can create some changable setting that can work for un-official api, maybe even a separate concrete class in the LLM folder

@parag477 is this ok for you?

[dartpain]

Think both should be fixed now in any case. I will also create a new issue to support vllm type of proxies properly