Characters other than English characters are incorrectly saved in the database

[TGIANN]

What happened?

Yesterday I updated to the latest artifact version and the names of the players with names with Turkish characters (ş,ü,ğ,ç etc) are not saved in the database

I typed "tgiann testsşğü" as a test and this is how it is saved

QBShared.FirstLatterCapitalize = function(text)
    local newText = ""
    local first = true
    for token in string.gmatch(text, "[^%s]+") do
        if first then
            first = false
            newText = string.gsub(string.lower(token), "^%l", string.upper)
        else
            newText = newText.." "..string.gsub(string.lower(token), "^%l", string.upper)
        end
    end
	return newText
end

Expected result

Proper registration of Turkish characters

Reproduction steps

1. Updated lastest version
2. I change my name and it registers incorrectly
3. Updating to one of the 4 months old versions
4. I change my name again and it saved normal

Importancy

Unknown

Area(s)

FiveM

Specific version(s)

Fivem 7290 | Game build 2802

Additional information

No response

[matehun00]

Not a Fivem issue

[TGIANN]

Not a Fivem issue

old version artifacts do not have this problem

[matehun00]

Not a Fivem issue

old version artifacts do not have this problem

Contact QB support then, I think they have a discord

[matehun00]

Could you at least give some code in text so we can reproduce it

[blattersturm]

Can you check if this worked in server version 7166? The only relevant change I can think of would be 7146cdad9f760caef9e14d2fc2544d3bb957eef9 which first showed up in 7168, but without knowing specific environment details (like what database client resource you're using) this is kind of difficult to investigate.

[TGIANN]

Can you check if this worked in server version 7166? The only relevant change I can think of would be 7146cda which first showed up in 7168, but without knowing specific environment details (like what database client resource you're using) this is kind of difficult to investigate.

I tested 7166 and 7168 on my own computer and everything is fine, then I upgraded to the latest version and everything is fine, but on my vds I have this problem

tested on vds 7166

7168 :(

windows version: Windows Server 2019 Datacanter mysql script: oxmysql 2.7.6 mariaDB

[V3SC]

v6.0.2 atop FXServer 7290

charset=utf8mb4_unicode_ci to the mysql connection string

[TGIANN]

v6.0.2 atop FXServer 7290

charset=utf8mb4_unicode_ci to the mysql connection string

I fixed the problem I updated the mariadb version and changed the function to this one but it's strange that what worked in older versions doesn't work in the new version

QBShared.FirstLatterCapitalize = function(text)
	local lower = string.lower(text)
	local capitalize = string.gsub(lower, "^%l", string.upper)
	return capitalize
end

[blattersturm]

Please don't close the issue solely because you fixed your problem. This is still a potential regression that should be investigated.

Does it still happen if you used the 'function' above?

[outsider31000]

yes we have this in redm too same for item names on the databse its including weird symbols

@blattersturm

[outsider31000]

artifacts version was the recommended in the cfx discord announcement before updating there was no issue

[Disquse]

I'm not sure if it's indeed related to https://github.com/citizenfx/fivem/commit/7146cdad9f760caef9e14d2fc2544d3bb957eef9, but maybe @jakub-cfx you can take a look?

[TGIANN]

Please don't close the issue solely because you fixed your problem. This is still a potential regression that should be investigated.

Does it still happen if you used the 'function' above?

I have not solved my problem, it still persists In version 7166 everything works properly

[jakub-cfx]

I'm not sure if it's indeed related to 7146cda, but maybe @jakub-cfx you can take a look?

Will definitely take a look, bit surprised that extra support ended in a regression, hopefully pinpointing where encodings diverge shouldn't take too long 🙏

[jakub-cfx]

@TGIANN would you mind posting the output of dism /online /get-intl on the affected VDS? should help with piecing together a repro scenario

[gottfriedleibniz]

@outsider31000 Since you are experiencing the same issue, are you capable of running something similar to what jakub posted? Also would you happen to know what version of Windows your server is running?

It also may help if the hex encodings of the two strings are provided (e.g., expected vs. actual).

[outsider31000]

seems to happen after A letter windows i am running pretty sure its windows 10

So far the ones I fixed manually they have not came back with those symbols so to run that code dont think will show me results I am still using the recommend build

those images I have shared were posted at aprox the same time this post was made. so its been some time. and I have not had reports of new issues like these. but will continue to report here.

[gottfriedleibniz]

Ah. ’ is a classic example of ' in UTF-8 being decoded as Windows-1252.

For reference, what version of Windows 10 are you using? You can do Windows key + R + winver. I am looking for the Version and Build strings.

[outsider31000]

here

[gottfriedleibniz]

Nice. Thanks.

For reference, both you and TGIANN seem to be using 1809. This may point to Windows builds before 1903 requiring their own special handling - will require testing.

[outsider31000]

thank you

[jakub-cfx]

@outsider31000 Could you please run dism /online /get-intl in an Administrator Command Prompt and paste the full output here? Additionally, it would be very helpful if you could share your database connection string with us (without the username and password of course, just the rest of the connection parameters) as well as the destination table's encoding/collation. In case you're using MariaDB like @TGIANN you can connect to the DB in HeidiSQL, right-click & Edit the table, and the window that pops up will show you the required details. I attached a few pictures to illustrate the process. With all these info we should be able to create a near perfect bug repro environment and find the root cause faster. Thank you 🙏

[outsider31000]

set mysql_connection_string "mysql://*********/vorpv2?connectTimeout=600000"

[outsider31000]

let me know if anything else

[jakub-cfx]

Tried to repro on English (US) Windows Server 2019 (build 17763.5329) with MariaDB 10.4.27, FXServer 7290 and oxymsql 2.7.6 just like the previous comments suggested, and single quotes ' as well as the aforementioned Turkish characters ş,ü,ğ,ç do get written to the database correctly. Tried writing them directly from hardcoded strings in server.lua as well as passing them as player input from a registered command (from latest prod client build, running as GTA b2802), still no repro unfortunately.

The last thing I can think of is that this is an issue on outdated Server 2019 installs, can you maybe try updating to the newest CU and see if the issue persists?

[outsider31000]

I have not had this issue with latest artifacts for some time so I'm guessing it's fixed. @jakub-cfx

[jakub-cfx]

I have not had this issue with latest artifacts for some time so I'm guessing it's fixed.

Thanks for the heads up, fingers crossed it stays that way 😊