bancho.py icon indicating copy to clipboard operation
bancho.py copied to clipboard

bug: could not send emoji due to mysql encoding

Open arily opened this issue 2 months ago • 0 comments

Describe the bug

unicode in MySQL defaults to utf8mb3, which can not store emoji correctly. anyone sending emojis in game will crash the packet handler. change column mail.msg to utf8mb4 can solve this problem.

-- alter existing table
ALTER TABLE `mail` CHANGE `msg` `archer(2048)` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

-- during creation
CREATE TABLE `mail` (
--  `id` int NOT NULL AUTO_INCREMENT,
--  `from_id` int NOT NULL,
--  `to_id` int NOT NULL,
  `msg` varchar(2048) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
--  `time` int DEFAULT NULL,
--  `read` tinyint(1) NOT NULL DEFAULT '0',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=106152 DEFAULT CHARSET=utf8mb3;

edit: upon further checking I found this line

Older versions of MySQL (prior to 8.0) typically used latin1 as the default character set, and utf8 (which is an alias for utf8mb3 and only supports 3-byte UTF-8, thus not full Unicode) was also common.

I believe this may not caused by MySQL but due to we explicitly used "utf8" charset. https://github.com/osuAkatsuki/bancho.py/blob/a66e4bdc4a38ac5cac1aece308c51817f5047c5c/migrations/base.sql#L6

To Reproduce

https://github.com/user-attachments/assets/61d7c95d-8da9-4ab1-8b0b-05b9ce641c5d

Expected behavior

should be able to send emoji correctly without crashing server.

bancho.py Version

3.5.0

Python Version

3.11.X (Default)

Relevant log output

File "/osu-server/bancho.py/app/api/domains/cho.py", line 261, in bancho_handler
    await packet.handle(player)
  File "/osu-server/bancho.py/app/api/domains/cho.py", line 1322, in handle
    await mail_repo.create(
  File "/osu-server/bancho.py/app/repositories/mail.py", line 62, in create
    rec_id = await app.state.services.database.execute(insert_stmt)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/osu-server/bancho.py/app/adapters/database.py", line 123, in execute
    rec_id = await self._database.execute(query, params)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/databases/core.py", line 199, in execute
    return await connection.execute(query, values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/databases/core.py", line 318, in execute
    return await self._connection.execute(built_query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/databases/backends/mysql.py", line 157, in execute
    await cursor.execute(query_str, args)
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/cursors.py", line 239, in execute
    await self._query(query)
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/cursors.py", line 457, in _query
    await conn.query(q)
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/connection.py", line 469, in query
    await self._read_query_result(unbuffered=unbuffered)
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/connection.py", line 683, in _read_query_result
    await result.read()
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/connection.py", line 1164, in read
    first_packet = await self.connection._read_packet()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/connection.py", line 652, in _read_packet
    packet.raise_for_error()
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/pymysql/protocol.py", line 219, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/pymysql/err.py", line 150, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.DataError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x80' for column 'msg' at row 1")
ERROR:    Exception in ASGI application

Additional context

No response

arily avatar Oct 16 '25 12:10 arily