bot
bot copied to clipboard
python-discord
Fix bugs with sending deleted messages to site
There are currently a couple of bugs with our sending of deleted messages to site.
Race condition with antispam:
bot_1 | 2022-05-29 09:41:44 | bot.exts.backend.error_handler | ERROR | API responded with 400 for command clean until: {'deletedmessage_set': [{'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {'id': ['deleted message with this ID already exists.']}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}]}.
To reproduce, spam !clean until <message_id> in a channel until you get the error. Antispam and the clean cog both try and delete the messages and upload them at the same time, causing the error.
A potential solution would be to allow two message deletion contexts to share a message on the site side. Not exactly sure how this would work.
Unable to delete webhooks
bot_1 | 2022-05-29 09:52:04 | bot.exts.backend.error_handler | ERROR | API responded with 400 for command clean until: {'deletedmessage_set': [{}, {}, {}, {'author': ['Invalid pk "871454771276550156" - object does not exist.']}]}.
To reproduce, try and delete a webhook using the clean command. It seems we just don't have handling for this.
Error sending embeds with null bytes
Error
postgres_1 | 2022-05-29 09:56:45.501 UTC [4417] ERROR: unsupported Unicode escape sequence at character 205
postgres_1 | 2022-05-29 09:56:45.501 UTC [4417] DETAIL: \u0000 cannot be converted to text.
postgres_1 | 2022-05-29 09:56:45.501 UTC [4417] CONTEXT: JSON data, line 1: {"type": "rich", "title":...
postgres_1 | 2022-05-29 09:56:45.501 UTC [4417] STATEMENT: INSERT INTO "api_deletedmessage" ("id", "author_id", "channel_id", "content", "embeds", "attachments", "deletion_context_id") VALUES (980409284938592286, 697133219862151208, 813116922118930459, '', ARRAY['{"type": "rich", "title": "hello\u0000"}']::jsonb[], '{}'::varchar(512)[], 4)
web_1 | Internal Server Error: /api/bot/deleted-messages
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
web_1 | return self.cursor.execute(sql, params)
web_1 | psycopg2.errors.UntranslatableCharacter: unsupported Unicode escape sequence
web_1 | LINE 1: ...697133219862151208, 813116922118930459, '', ARRAY['{"type": ...
web_1 | ^
web_1 | DETAIL: \u0000 cannot be converted to text.
web_1 | CONTEXT: JSON data, line 1: {"type": "rich", "title":...
web_1 |
web_1 |
web_1 | The above exception was the direct cause of the following exception:
web_1 |
web_1 | Traceback (most recent call last):
web_1 | File "/usr/local/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
web_1 | response = get_response(request)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
web_1 | response = wrapped_callback(request, *callback_args, **callback_kwargs)
web_1 | File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/views.py", line 67, in sentry_wrapped_callback
web_1 | return callback(request, *args, **kwargs)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
web_1 | return view_func(*args, **kwargs)
web_1 | File "/usr/local/lib/python3.9/site-packages/rest_framework/viewsets.py", line 125, in view
web_1 | return self.dispatch(request, *args, **kwargs)
web_1 | File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 509, in dispatch
web_1 | response = self.handle_exception(exc)
web_1 | File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 469, in handle_exception
web_1 | self.raise_uncaught_exception(exc)
web_1 | File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
web_1 | raise exc
web_1 | File "/usr/local/lib/python3.9/site-packages/rest_framework/views.py", line 506, in dispatch
web_1 | response = handler(request, *args, **kwargs)
web_1 | File "/usr/local/lib/python3.9/site-packages/rest_framework/mixins.py", line 19, in create
web_1 | self.perform_create(serializer)
web_1 | File "/usr/local/lib/python3.9/site-packages/rest_framework/mixins.py", line 24, in perform_create
web_1 | serializer.save()
web_1 | File "/usr/local/lib/python3.9/site-packages/rest_framework/serializers.py", line 205, in save
web_1 | self.instance = self.create(validated_data)
web_1 | File "/app/pydis_site/apps/api/serializers.py", line 126, in create
web_1 | DeletedMessage.objects.create(
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
web_1 | return getattr(self.get_queryset(), name)(*args, **kwargs)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 447, in create
web_1 | obj.save(force_insert=True, using=self.db)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 753, in save
web_1 | self.save_base(using=using, force_insert=force_insert,
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 790, in save_base
web_1 | updated = self._save_table(
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 895, in _save_table
web_1 | results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/base.py", line 933, in _do_insert
web_1 | return manager._insert(
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
web_1 | return getattr(self.get_queryset(), name)(*args, **kwargs)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/query.py", line 1254, in _insert
web_1 | return query.get_compiler(using=using).execute_sql(returning_fields)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
web_1 | cursor.execute(sql, params)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 98, in execute
web_1 | return super().execute(sql, params)
web_1 | File "/usr/local/lib/python3.9/site-packages/sentry_sdk/integrations/django/__init__.py", line 493, in execute
web_1 | return real_execute(self, sql, params)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
web_1 | return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
web_1 | return executor(sql, params, many, context)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
web_1 | return self.cursor.execute(sql, params)
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
web_1 | raise dj_exc_value.with_traceback(traceback) from exc_value
web_1 | File "/usr/local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
web_1 | return self.cursor.execute(sql, params)
web_1 | django.db.utils.DataError: unsupported Unicode escape sequence
web_1 | LINE 1: ...697133219862151208, 813116922118930459, '', ARRAY['{"type": ...
web_1 | ^
web_1 | DETAIL: \u0000 cannot be converted to text.
web_1 | CONTEXT: JSON data, line 1: {"type": "rich", "title":...
web_1 |
web_1 | "POST /api/bot/deleted-messages HTTP/1.1" 500 235164
bot_1 | 2022-05-29 09:56:45 | charset_normalizer | TRACE | Detected declarative mark in sequence. Priority +1 given for utf_8.
bot_1 | 2022-05-29 09:56:45 | charset_normalizer | TRACE | Code page utf_8 is a multi byte encoding table and it appear that at least one character was encoded using n-bytes.
bot_1 | 2022-05-29 09:56:45 | charset_normalizer | TRACE | utf_8 passed initial chaos probing. Mean measured chaos is 0.000000 %
bot_1 | 2022-05-29 09:56:45 | charset_normalizer | TRACE | We detected language [('English', 1.0), ('Indonesian', 1.0), ('Simple English', 1.0), ('Dutch', 1.0), ('Swedish', 1.0)] using utf_8
bot_1 | 2022-05-29 09:56:45 | charset_normalizer | DEBUG | Encoding detection: utf_8 is most likely the one.
bot_1 | 2022-05-29 09:56:45 | bot.exts.backend.error_handler | WARNING | API responded with 500 for command clean until
!int eval
embed = discord.Embed(title="hello\x00")
await ctx.send(embed=embed)
And try and clean the resulting message. We currently strip null bytes from message content, but not from any embed contents. https://github.com/python-discord/bot/blob/c29434b964e87743869723da51a7908be7cc7156/bot/exts/moderation/modlog.py#L63
Unknown issue
This is #BOT-34M on sentry. So far we've not managed to work out what the cause of this is or how to reproduce it. #2180 should help us debug this further.
API responded with 400 for command cleanban: {'deletedmessage_set': [{}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {}, {'embeds': {'0': ["Tag embed must contain one of the fields {'video', 'title', 'description', 'image', 'fields'}."]}}]}.
