bug: could not send emoji due to mysql encoding
Describe the bug
unicode in MySQL defaults to utf8mb3, which can not store emoji correctly.
anyone sending emojis in game will crash the packet handler.
change column mail.msg to utf8mb4 can solve this problem.
-- alter existing table
ALTER TABLE `mail` CHANGE `msg` `archer(2048)` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
-- during creation
CREATE TABLE `mail` (
-- `id` int NOT NULL AUTO_INCREMENT,
-- `from_id` int NOT NULL,
-- `to_id` int NOT NULL,
`msg` varchar(2048) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL,
-- `time` int DEFAULT NULL,
-- `read` tinyint(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=106152 DEFAULT CHARSET=utf8mb3;
edit: upon further checking I found this line
Older versions of MySQL (prior to 8.0) typically used latin1 as the default character set, and utf8 (which is an alias for utf8mb3 and only supports 3-byte UTF-8, thus not full Unicode) was also common.
I believe this may not caused by MySQL but due to we explicitly used "utf8" charset. https://github.com/osuAkatsuki/bancho.py/blob/a66e4bdc4a38ac5cac1aece308c51817f5047c5c/migrations/base.sql#L6
To Reproduce
https://github.com/user-attachments/assets/61d7c95d-8da9-4ab1-8b0b-05b9ce641c5d
Expected behavior
should be able to send emoji correctly without crashing server.
bancho.py Version
3.5.0
Python Version
3.11.X (Default)
Relevant log output
File "/osu-server/bancho.py/app/api/domains/cho.py", line 261, in bancho_handler
await packet.handle(player)
File "/osu-server/bancho.py/app/api/domains/cho.py", line 1322, in handle
await mail_repo.create(
File "/osu-server/bancho.py/app/repositories/mail.py", line 62, in create
rec_id = await app.state.services.database.execute(insert_stmt)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/osu-server/bancho.py/app/adapters/database.py", line 123, in execute
rec_id = await self._database.execute(query, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/databases/core.py", line 199, in execute
return await connection.execute(query, values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/databases/core.py", line 318, in execute
return await self._connection.execute(built_query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/databases/backends/mysql.py", line 157, in execute
await cursor.execute(query_str, args)
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/cursors.py", line 239, in execute
await self._query(query)
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/cursors.py", line 457, in _query
await conn.query(q)
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/connection.py", line 469, in query
await self._read_query_result(unbuffered=unbuffered)
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/connection.py", line 683, in _read_query_result
await result.read()
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/connection.py", line 1164, in read
first_packet = await self.connection._read_packet()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/aiomysql/connection.py", line 652, in _read_packet
packet.raise_for_error()
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/pymysql/protocol.py", line 219, in raise_for_error
err.raise_mysql_exception(self._data)
File "/root/.cache/pypoetry/virtualenvs/bancho-py-bdlVfY0W-py3.11/lib/python3.11/site-packages/pymysql/err.py", line 150, in raise_mysql_exception
raise errorclass(errno, errval)
pymysql.err.DataError: (1366, "Incorrect string value: '\\xF0\\x9F\\x98\\x80' for column 'msg' at row 1")
ERROR: Exception in ASGI application
Additional context
No response