Sending a very specific strings to discord webhook causes it to handle it as ISO-8859-9 instead of UTF-8

[linus-jansson]

Description

I was sending text files to a discord webhook on my server and found out that something wasn't right when I added å, ä or ö to the webhook. I experimented a bit and very specific sentences seems to break the encoding. If for example sending åäö hello world växte ölen 28 very long stuff using the steps to reproduce it works. if writing Word "växte" was found in 28 tries it breaks. I cannot seem to find a pattern in what causes it.

Steps to Reproduce

echo "Word "växte" was found in 28 tries" > /tmp/test.txt && hexdump /tmp/test.txt -C && curl -X POST \
        -H "Content-Type: multipart/form-data; charset=utf-8" \
        -F 'payload_json={"content":"invalid charset in preview!"}' \
        -F "file=@/tmp/test.txt" \
        $DISCORD_WEBHOOK_URL

or using a github action

name: Test encoding

on:
  push:
    branches:
      - main

jobs:
  run-main:
    runs-on: ubuntu-latest

    steps:

    - name: Webhook DEBUG example file
      run: |
        echo 'Word "växte" was found in 28 tries' > /tmp/test.txt && curl -X POST \
        -H "Content-Type: multipart/form-data; charset=utf-8" \
        -F 'payload_json={"content":"DEBUG FILE!"}' \
        -F "file=@/tmp/test.txt" \
        ${{ secrets.DISCORD_WEBHOOK }}

Expected Behavior

It should properly parse the file as utf8.

Current Behavior

Without explicitly writing a utf8 BOM to the file it handles the file marked in the echo command above as ISO-8859-9 when the response from the discord webhook returns.

{
  "type": 0,
  "channel_id": "<RADICATED>",
  "content": "invalid charset in preview!",
  "mentions": [],
  "mention_roles": [],
  "attachments": [
    {
      "id": "1255272562884083732",
      "filename": "test.txt",
      "size": 34,
      "url": "<RADICATED>",
      "proxy_url": "<RADICATED>",
      "content_type": "text/plain; charset=ISO-8859-9",
      "content_scan_version": 0
    }
  ],
  "embeds": [],
  "timestamp": "2024-06-25T21:25:00.809000+00:00",
  "edited_timestamp": null,
  "flags": 0,
  "components": [],
  "id": "<RADICATED>",
  "author": {
    "id": "1253455521122488321",
    "username": "<RADICATED>",
    "avatar": null,
    "discriminator": "0000",
    "public_flags": 0,
    "flags": 0,
    "bot": true,
    "global_name": null,
    "clan": null
  },
  "pinned": false,
  "mention_everyone": false,
  "tts": false,
  "webhook_id": "<RADICATED>"
}

Screenshots/Videos

No response

Client and System Information

• Github actions
• WSL 2 on windows 11 using ubuntu 20.04
• Curl