Make Char.isLower and Char.isUpper Unicode-aware

[Janiczek]

This allows people with non-ASCII alphabets work with Char.isLower and Char.isUpper. Uses toUpper and toLower underneath, which use Javascript's String.prototype.toLower/UpperCase().

The second condition in the functions is there to distinguish between characters that have an upper/lower-case pairing, and those that don't ('0' == Char.toLower '0' but we don't want isLower '0' to be true).

EDIT: I can't update code of this PR anymore; there is #1138 with a fix for the == and /= import.

[Janiczek]

Is related to #385.

[drathier]

What's considered an uppercase character depends on your locale. This PR is still a major improvement.

Related to #942.

[evancz]

For future reference, the toLocaleUpperCase function talks about cases where this will break:

The toLocaleUpperCase() method returns the value of the string converted to upper case according to any locale-specific case mappings. toLocaleUpperCase() does not affect the value of the string itself. In most cases, this will produce the same result as toUpperCase(), but for some locales, such as Turkish, whose case mappings do not follow the default case mappings in Unicode, there may be a different result.

So it seems that toUpperCase() is a pure function, but toLocaleUpperCase() is not. My instinct is that the "correct" version of this function takes Language as an argument. (Not sure if the idea of "locale" is better. Maybe it is geographical? Maybe there is some standards body that defines locales?)

I do not want us to theorize about these things here. The next step is to find nice links that describe:

1. How "upper case" is defined by unicode. Is there a big table somewhere?
2. How a "locale" is defined and who manages that. Are there "new locales" if human culture changes? Who captures that, and how do browsers know about it?

I would prefer to understand the problem more completely before changing things.

[Janiczek]

From my cursory googling and research:

1. How is "upper case" defined?

I think this FAQ is the link you want.

In short, yes, there is a big table. Three, in fact.

1. ftp://ftp.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt
2. ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt
3. ftp://ftp.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt (GitHub or Markdown doesn't support FTP links :man_shrugging: )

Here is the relevant section of standard.

It has some sense of inter-version stability between the Unicode versions.

2. How is "locale" defined?

Again, an Unicode FAQ; and this time there's a whole homepage.

You can download the current version, there are a lot of XML files inside with various data (casing of dates / languages / ..., etc.), to be interpreted according to LDML.

They are also transformed from the XML into JSON, which might be a better fit for Elm?

[Janiczek]

We might try to be extra-pure and host the big table, in Elm format, ourselves, but that would make elm/core very big, I imagine. The browser already has that cached in the form of .toLowerCase().

The .toLocaleLowerCase() functions would benefit from the Language argument (to become pure), but I wonder if it's important. In what situation would the function start misbehaving? (From the top of my head, computer location changing? System settings changing?) And is it important, would it affect the user somehow?

I mean, even Date, when toStringed, will show different things on different machines, based on your timezone.

main : Html msg
main =
    "2018-05-10"
        |> Date.fromString
        |> toString
        |> Html.text

shows Ok <Thu May 10 2018 02:00:00 GMT+0200 (Central European Summer Time)> on my machine. (Ellie) It will presumably show something different on yours. It's not pure. Is that problematic?

[gormonn]

I ran into the same issue when doing the exercise in the forms section, namely checking the uppercase password. (btw, it's sad to see such behavior even before deep acquaintance with the language.)

At first glance, I thought this was a serious omission.

However, then I wondered if it was worth letting users set their passwords to Unicode. On the one hand, this password be more resistant to brute force. On the other hand, it may be inconvenient for the user if the device does not have a specific locale.

However, in any case, this is not decided at the stage of front-end approval, but much earlier. Thus, this limitation may cause frustration for developers from regions other than English. And negatively affects the use of Elm as the main front-end stack in the Enterprise environment. This means about the popularity and development of the language.

But I believe that such an annoying flaw will still not be a problem for Elm.

Meanwhile. I’m very curious if I’m stuck in mind trap just because Elm doesn’t support my native language well. There may be other uses for Unicode and Char.isUpper that we are not aware of. So write if you know this.

[avh4]

FYI, this package can currently be used to deal with unicode strings: https://package.elm-lang.org/packages/BrianHicks/elm-string-graphemes/latest/

[sagehane]

https://github.com/elm/core/blob/e47edebc0240dc0b840875d7714f9393e0b3f45d/src/Char.elm#L84-L85

Wouldn't char /= Char.toLower char suffice? If a character does not equal its lowercase counterpart, it must be uppercase.

A would become a, True a would become a, False 0 would become 0, False

Same applies to isLower.

(I tried using code comments, didn't work, idk why)

Edit: The one scenario where this might make a difference is if there's a "middle case" character that has both an upper and a lower case variant. But I don't think such a character exists, and even if it does, should isUpper return a True or False in such a scenario?

[miniBill]

@sagehane

> let s = Char.fromCode 453 in (s, Char.toUpper s, Char.toLower s)
('Dž','DŽ','dž') : ( Char, Char, Char )

[sagehane]

@miniBill, good to know. So, would you argue that 'Dž'.isLower() should return true, or false? I feel like if a character has an uppercase form, it should be considered a lowercase character. That is, I disagree with the current code.

[miniBill]

Dž is neither lowercase nor uppercase, according to Unicode

https://ellie-app.com/gBmgVVFzhbRa1 this contains a table of all the 1441 codepoints that give wrong results with the current proposal

I personally think the proposal is the best compromise between accuracy and size/speed. Getting better results belongs in external packages (like elm-unicode)

[rupertlssmith]

This PR causes a compile error when it is used: https://github.com/elm-janitor/apply-patches/issues/1

[Janiczek]

@rupertlssmith I can't edit this PR's code anymore, see #1138 for compilable code.