kotter icon indicating copy to clipboard operation
kotter copied to clipboard

Fix busted unicode (emoji? chinese characters?) support

Open bitspittle opened this issue 3 years ago • 4 comments

Well, nuts. String.length miscounts unicode characters and we might be using it everywhere.

Something like this should render incorrectly

bordered {
   textLine("Hello")
   textLine("Oh my stars ⭐⭐⭐ this just screws it up")
   textLine("Goodbye") 
}

Add stars to test cases, and probably use text.codePoints.count() instead of text.length. Audit all the code...

bitspittle avatar Apr 16 '22 05:04 bitspittle

Not a blocker for 1.0.

It's not great, sure, but emojis (many of them anyway) work, and it's in some ways more cosmetic than fundamental. Plus, we can always fix it in the future and the change should be backwards compatible with older codebases.

bitspittle avatar Oct 08 '22 03:10 bitspittle

FYI, this is an especially tricky/vexing issue—the Unicode standard gets updated all the time, and making sure you measure emojis correctly (especially combining sequences like person + hat + skin color) usually means interoperating with a native library. And then there’s East Asian character sets, but at least those don’t change multiple times a year.

mstrobel avatar Jan 30 '23 00:01 mstrobel

Even worse than I thought 😂 I have given up on this one at the moment. I looked briefly into the places I was using str.length and I recall some case being tricky to handle. I probably should have left a note on this bug what that was...

But yeah I guess first pass would be finding the right third party library to include which did the counting for me.

bitspittle avatar Jan 30 '23 01:01 bitspittle

Report from a user. This looks like it's actively causing a program when trying to repaint Chinese characters:

Not working: https://user-images.githubusercontent.com/43705986/228932658-9ebe0622-0bf2-4157-b193-c6008eaa01e9.mp4

Working: https://user-images.githubusercontent.com/43705986/228932683-3fb2a15a-4e70-4e12-91a6-8a485ad23ae8.mp4

Code to reproduce:

var success by liveVarOf<Boolean?>(null)
var errorMsg by liveVarOf<String?>(null)

val blinkTotalLen = 1.seconds
val blinkLen = 250.milliseconds
var blinkOn by liveVarOf(false)

section {
    textLine("Please input:")
    items.forEach { codeData -> textLine("  $codeData") }
    text("> "); input()
    success?.let {
        p {
            if (!it)  {
                red { text("×") }; text(" ")
                scopedState {
                    red(isBright = true) {
                        if (blinkOn) invert()
                        textLine("LKJLKJLJLKJLKLHJKHJSLKDJLKSJDLKSJDLKSNM<ADLKSJKDLJSLKDJKLAJSLKAJKLJLKSKLSHFKLSJDKLAJSDLKAJDKLSJDLKSFLJKSHKLFHKLSJDLKAJSLKAJLKJKLSFKLHSKLFJKLS")
//                            textLine("错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误错误")
                        textLine("caused by: ${errorMsg?: ""}")
                    }
                }
            }
        }
    }
}.runUntilSignal {
    onInputEntered {
        if(input.any { !it.isDigit() }) {
            errorMsg = "error "
            success = false
            rejectInput()
            var blinkCount = blinkTotalLen.inWholeMilliseconds / blinkLen.inWholeMilliseconds
            addTimer(blinkLen, repeat = true) {
                blinkOn = !blinkOn
                blinkCount--
                if (blinkCount == 0L) {
                    repeat = false
                }
            }
        }
    }
}

bitspittle avatar Mar 30 '23 18:03 bitspittle