f2e-spec
f2e-spec copied to clipboard
Accented characters sorting problem
Now: ABCDEFGHIJKLMNOPQRSTZUVWXYZÁÖÜ... It should be: AÁBCDEÉFGHIÍJKLMNOÓÖŐPQRSTUÚÜŰVQXYZ
This has to do with the way UTF-8 works. :tired_face: UTF-8 is just ascii (7 bits) so it sorts in that order, only when the first bit is 1 sonething special happens. the first byte is the header that shows how much "bytes" long the unicode character is, so: [11100000] means that you get 3 bytes that form a number, anyway this guy can explain it better than me: https://www.youtube.com/watch?v=MijmeoH9LT4 so because all (accented) unicode characters are higher than the ascii symbols they appear higher in the list when sorting. this isn't really a performous specific bug i'm afraid, more a general problem with the way computers work.
Still, note that there are conversion tables in Unicode from accented to unaccented characters, so the sorting could be performed using a hidden simplified name.
See e.g. https://pypi.python.org/pypi/Unidecode (for Python).
Incidentally, Spanish would too benefit from this.
Yeah... bit our lazy byte-wise sorting doesn't take that into account 😅😅
So what will we do then?
is it worth the effort to include an unicode conversion table?
A lot of countries use them. Would be nice to have :)
is this still an issue with current master (using ICU for text processing now)? @Lord-Kamina
ICU should actually fix this, so we should test it and then close this issue.
I know it has the option to include accents in sorting but I don't think it does it by default. So we should test.
Also, I don't know think we're currently using ICU in sorting
On March 20, 2018 at 04:38:51, niek nooijens ([email protected]) wrote:
ICU should actually fix this, so we should test it and then close this issue.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/performous/performous/issues/187#issuecomment-374502851, or mute the thread https://github.com/notifications/unsubscribe-auth/AGJfLIcUG5Tc0l-X7WAcLxal_rqjjmmPks5tgLIKgaJpZM4I2sM0 .
I just confirmed Sorting is not working as explained above.
I have 3 songs with artist:
- Aoulevard des airs
- Boulevard des airs
- Áoulevard des airs
it's sorted like the list above while being sorted on artist Searching on the accented characters works though (#162)
I'll look into how I can fix it later.
On March 29, 2018 at 16:05:10, Arjan Spieard ([email protected]) wrote:
I just confirmed Sorting is not working as explained above.
I have 3 songs with artist:
- Aoulevard des airs
- Boulevard des airs
- Áoulevard des airs
it's sorted like the list above while being sorted on artist Searching on the accented characters works though (#162 https://github.com/performous/performous/issues/162)
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/performous/performous/issues/187#issuecomment-377339871, or mute the thread https://github.com/notifications/unsubscribe-auth/AGJfLAWfVY0XJEgbdyz4FG5DIaY71XpKks5tjTBlgaJpZM4I2sM0 .
I don't think this was ever looked into? @Lord-Kamina
Nope, but ICU has a collator that can be set according to different language rules. I can't look it up now though because I'm at work
So the basic idea is that you sort by keys that are plain ASCII and formed by a collate function out of the original UTF-8 so that it sorts correctly using bytewise comparisons (e.g. using std::map with std::string collate keys). All very easy so far, and probably already be implemented (for case-insensitive ordering in English). A bit of a problem is that all this should follow the locale that Performous is running in because the order is different in different languages (e.g. in Finnish Ä and Ö come after Z and are not equal to A and O).
I'll have to check the documentation but I'm fairly sure it's already implemented
http://userguide.icu-project.org/collation
ICU has the API, we just need to implement it.
IMO, it should be part of #524
I wouldn't include it in #524 since its a complete new feature, toggling language
I wouldn't include it in #524 since its a complete new feature, toggling language
I meant because, IMO at least, we should sort as-per the language performous is currently running in.