UnicodeEncodeError: 'latin-1' codec can't encode character '\u2026' in position 512: ordinal not in range(256)

[Strykar]

Tested on Arch Linux, kernel 6.2.1 and Python 3.10.9

Mar 08 20:25:36 phragyaan systemd[1]: Started OpenAI Assistant (ChatGPT) IRC bot.
Mar 08 20:25:55 phragyaan python[332238]: None
Mar 08 20:25:55 phragyaan python[332238]: -> NICK ChatGPT
Mar 08 20:25:55 phragyaan python[332238]: -> USER text-davinci-002-render * * :OpenAI Assistant
Mar 08 20:25:55 phragyaan python[332238]: -> MODE ChatGPT +B
Mar 08 20:25:55 phragyaan python[332238]: -> JOIN #ChatGPT
Mar 08 20:25:55 phragyaan python[332238]: -> JOIN #wakka
Mar 08 20:25:55 phragyaan python[332238]: [#wakka] <Strykar> ChatGPT: hi
Mar 08 20:25:55 phragyaan python[332238]: -> PRIVMSG #wakka :Strykar: hold on, I'm thinking..
Mar 08 20:25:55 phragyaan python[332238]: Traceback (most recent call last):
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/local/bin/chatgpt-irc.py", line 240, in <module>
Mar 08 20:25:55 phragyaan python[332238]:     asyncio.run(main_loop(**options))
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
Mar 08 20:25:55 phragyaan python[332238]:     return loop.run_until_complete(main)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
Mar 08 20:25:55 phragyaan python[332238]:     return future.result()
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/local/bin/chatgpt-irc.py", line 230, in main_loop
Mar 08 20:25:55 phragyaan python[332238]:     messages = chatgpt.prompt(prompt)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/local/bin/chatgpt-irc.py", line 62, in prompt
Mar 08 20:25:55 phragyaan python[332238]:     query = requests.request("POST", url, headers=headers, data=payload)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
Mar 08 20:25:55 phragyaan python[332238]:     return session.request(method=method, url=url, **kwargs)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
Mar 08 20:25:55 phragyaan python[332238]:     resp = self.send(prep, **send_kwargs)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
Mar 08 20:25:55 phragyaan python[332238]:     r = adapter.send(request, **kwargs)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
Mar 08 20:25:55 phragyaan python[332238]:     resp = conn.urlopen(
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
Mar 08 20:25:55 phragyaan python[332238]:     httplib_response = self._make_request(
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
Mar 08 20:25:55 phragyaan python[332238]:     conn.request(method, url, **httplib_request_kw)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
Mar 08 20:25:55 phragyaan python[332238]:     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/http/client.py", line 1282, in request
Mar 08 20:25:55 phragyaan python[332238]:     self._send_request(method, url, body, headers, encode_chunked)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/http/client.py", line 1323, in _send_request
Mar 08 20:25:55 phragyaan python[332238]:     self.putheader(hdr, value)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 224, in putheader
Mar 08 20:25:55 phragyaan python[332238]:     _HTTPConnection.putheader(self, header, *values)
Mar 08 20:25:55 phragyaan python[332238]:   File "/usr/lib/python3.10/http/client.py", line 1255, in putheader
Mar 08 20:25:55 phragyaan python[332238]:     values[i] = one_value.encode('latin-1')
Mar 08 20:25:55 phragyaan python[332238]: UnicodeEncodeError: 'latin-1' codec can't encode character '\u2026' in position 512: ordinal not in range(256)
Mar 08 20:25:55 phragyaan systemd[1]: chatgpt.service: Main process exited, code=exited, status=1/FAILURE
Mar 08 20:25:55 phragyaan systemd[1]: chatgpt.service: Failed with result 'exit-code'.

[troydieter]

Same here @Strykar - any thoughts?

[moonlightz]

The thoughts are that this piece of code will have this error and the author refuses to fix it and always keep closing git issues without giving further details. The only fix is to try fix it ourselves. It is obviously a bug about handling text... or I meaning, handling everything because when and error is about to happen, there are no measures to handle them. The code might work for the author and some but obviously there is something is missing.

[rj1]

does running the script w/ PYTHONIOENCODING=utf8 prepended fix it for you?

e.g. PYTHONIOENCODING=utf8 python chatgpt-irc.py

or better yet, maybe you need to set your shell environment to use utf-8.

[Strykar]

Shell always used UTF-8:

$ locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

$ PYTHONIOENCODING=utf8 /usr/bin/python /usr/local/bin/chatgpt-irc.py
None
-> NICK ChatGPT
-> USER text-davinci-002-render * * :OpenAI Assistant
-> MODE ChatGPT +B
-> JOIN #ChatGPT
-> JOIN #wakka
[#wakka] <Strykar> ChatGPT: hi
-> PRIVMSG #wakka :Strykar: hold on, I'm thinking..
Traceback (most recent call last):
  File "/usr/local/bin/chatgpt-irc.py", line 240, in <module>
    asyncio.run(main_loop(**options))
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/usr/local/bin/chatgpt-irc.py", line 230, in main_loop
    messages = chatgpt.prompt(prompt)
  File "/usr/local/bin/chatgpt-irc.py", line 62, in prompt
    query = requests.request("POST", url, headers=headers, data=payload)
  File "/usr/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3.10/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.10/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1323, in _send_request
    self.putheader(hdr, value)
  File "/usr/lib/python3.10/site-packages/urllib3/connection.py", line 224, in putheader
    _HTTPConnection.putheader(self, header, *values)
  File "/usr/lib/python3.10/http/client.py", line 1255, in putheader
    values[i] = one_value.encode('latin-1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u2026' in position 512: ordinal not in range(256)

[Jurek-Raben]

This is a copy-paste issue. For exmaple in firefox inspector, firfst switch to raw/ascii view of the token and cookie data, and then copy it. Because Firefox annoyingly shortens the data in the default view.

[rkarlsba]

If the application can't handle invalid input, it's surely a bug in the application ;)