Multibyte characters mangled on macOS

[mislav]

Directly after a pty.Open(), writing some multibyte characters such as 小明 and then a Delete key (0x7f) to the "slave" end, then reading from the "master" end results in a mangled final character. https://gist.github.com/mislav/82c3e2cfa2180c23201a523078b07f07

Expectation: the Delete key causes the whole last character to be deleted, not just its final byte.

My colleague @tibbes points out that Darwin has a IUTF8 input flag that can be set on (pseudo) terminals: https://github.com/apple/darwin-xnu/blob/2ff845c2e033bd0ff64b5b6aa6063a1f8f65aa32/bsd/sys/termios.h#L133

Indeed, doing this fixes the issue:

settings.Iflag |= 0x4000 // IUTF8
termios.Tcsetattr(pts.Fd(), termios.TCSANOW, &settings)

Perhaps this library could set IUTF8 by default? I'm not sure if there will be any negative side-effects to that, though. I haven't tested that approach in production.

[creack]

I would be inclined to indeed set the flag by default, but it requires some exploration to make sure it doesn't break existing code bases.

cc @kr, any thoughts on this?

[kr]

I would be inclined to indeed set the flag by default, but it requires some exploration to make sure it doesn't break existing code bases.

cc @kr, any thoughts on this?

I agree. No additional insight or information from me, unfortunately.

Seems useful, hopefully it won't break much!