First multi-byte character not being read correctly when run on the client side of a rlogin connection

[rphaniram]

On Solaris 11, ksh93 does not read the first multibyte character completely with rlogin(1).

To reproduce:

rlogin XXX.XXX.XXX.XXX

Password: Last login: Wed Aug 13 03:22:07 2019 from YYY.YYY.YYY.YYY Oracle Corporation SunOS 5.11 11.3 July 2017

locale

LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_ALL=

ksh

¿¿¿¿ <--- input "¿¿¿¿¿". The first character is gone.

ksh: ¿¿¿¿: not found [No such file or directory]

The character code for UTF-8 is as below: ¿:E38182 ¿:E38184 ¿:E38186 ¿:E38188 ¿:E3818A

Given that we have other inter-dependencies, not in a position to move to a later version of ksh93 at this point in time. Also, not certain the issue is fixed in the latest version.

Any pointers to fix patch/Workaround is much appreciated.

Hence, looking for issue resolution.

[krader1961]

Are you saying this works correctly with the ksh93u+ release (from 12 years ago) but not the current version? You did not tell us which ksh version you are using.

What does rlogin have to do with this issue? It seems like rlogin should be irrelevant. Can you explain how others can reproduce this problem? Since your problem description involves rlogin it is unclear whether the locale you documented is on the local or remote system or both.

[krader1961]

Also, what ksh statement results in the ksh: ¿¿¿¿: not found [No such file or directory] output?

[rphaniram]

As mentioned in the Subject line, have tried out on 2016-01-10-beta and version sh (AT&T Research) 93u+ 2012-08-01

Issue is seen on both.

I am NOT sure if it exists on the latest version but not in a position to try that out due to other dependencies.

Not sure if rlogin is irrelevant. However, with the same locale settings, we don't see the issue with telnet/ssh etc... Also, the locale that is documented is on the remote system to which we 'rlogin'.

Steps to reproduce:

To reproduce: [email protected]> rlogin XXX.XXX.XXX.XXX Password: Last login: Wed Aug 13 03:22:07 2019 from YYY.YYY.YYY.YYY Oracle Corporation SunOS 5.11 11.3 July 2017 [email protected]> locale LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_ALL= [email protected]> ksh [email protected]> ¿¿¿¿ <--- input "¿¿¿¿¿". The first character is gone. ksh: ¿¿¿¿: not found [No such file or directory] [email protected]>

[krader1961]

@rphaniram FWIW, when reporting Unicode related issues you should use the U+3042 notation for a code point rather than the E38182 UTF-8 hex sequence for the codepoint. It wasn't until now that I realized what you were trying to say. Correct me if I'm wrong but are you saying that after starting an interactive ksh shell over a rlogin session typing Unicode code points U+3042, U+3044, U+3046, U+3048, and U+3050 results in one of the characters not being displayed? I cannot reproduce that problem.

I don't have any systems that allow rlogin. But in a terminal on macOS, Fedora, OpenSuse, etc. and using ssh, and with the same locale, I have no problem entering that sequence of characters and seeing each one correctly displayed. For example:

21:16 macpro ~ > /bin/ksh
KSH PROMPT:1: あいうえぐ
/bin/ksh: あいうえぐ: not found
KSH PROMPT:2: 

That includes OpenIndiana 5.11 (a Solaris fork).

If this only affects rlogin sessions I would look at the stty -a output when it works compared to when it doesn't work. Also checking the output of set -o to check if the shells are configured the same would be a good idea.

So we're still in a "needs more information" condition.

[rphaniram]

@krader1961 Yes. Your description of the problem statement is accurate. And, the issue is seen when logged in through 'rlogin'. And, yes, we do not see the issue with same locale and a login through either ssh/telnet.

Sure. I shall try to get the 'stty -a' and 'set -o' o/p as requested.

Thanks for looking into the issue.

[krader1961]

I am very confident this has nothing to do with ksh per se. Mostly because neither of us sees the problem when rlogin is not part of the equation. It is likely that rlogin is broken or your ksh startup scripts are consuming one of the interactive bytes. I do not expect the output of stty -a or set -o to reveal any interesting differences.

[rphaniram]

Please find the data requested. Actually, there seems to be little or no difference between the rlogin and telnet/ssh ksh environment/terminal options.

1. stty -a output when it works(telnet/ssh) compared to when it doesn't work(rlogin).

# diff set-o_rlogin.txt set-o_telnet.txt
14c14
<opost -olcuc onlcr -ocrnl -onocr -onlret -ofill -ofdel
---
> opost -olcuc onlcr -ocrnl -onocr -onlret -ofill -ofdel tab3
                                                         ^^^^

Additionally, "tab3" is set during a telnet session.

Hence, we tested with "tab3" set too but the problem persists. 'tab3' does not seem to make any difference and looks unrelated to the issue

# rlogin XXX.XXX.XXX.XXX -l root
Password:
Last login: Tue Sep 3 00:09:06 2019 from XXX.XXX.XXX.YYY
Oracle Corporation SunOS 5.11 11.4 August 2019
# ksh
# stty -a
speed 38400 baud;
rows = 27; columns = 89; ypixels = 0; xpixels = 0;
csdata?
eucw 1: 0: 0: 0, scrw 1: 0: 0: 0
intr = ^ c; quit = ^ \; erase = ^ ?; kill = ^ u;
eof = ^ d; eol = <undef>; eol2 = <undef>; swtch = <undef>;
start = ^ q; stop = ^ s; susp = ^ z; dsusp = ^ y;
rprnt = ^ r; flush = ^ o; werase = ^ w; lnext = ^ v;
-parenb -parodd cs8 -cstopb -hupcl cread -clocal -loblk -crtscts -crtsxoff
-parext
-ignbrk brkint ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl -iuclc
ixon -ixany -ixoff imaxbel
isig icanon -xcase echo echoe echok -echonl -noflsh
-tostop echoctl -echoprt echoke -defecho -flusho -pendin iexten
opost -olcuc onlcr -ocrnl -onocr -onlret -ofill -ofdel
# &#12356;&#12288;&#12288;&#12288;&#12288;&#12288; &#8592; "&#12354;&#12356;"
is entered
ksh: i: not found [No such file or directory]
# stty tab3
# stty -a
speed 38400 baud;
rows = 27; columns = 89; ypixels = 0; xpixels = 0;
csdata?
eucw 1: 0: 0: 0, scrw 1: 0: 0: 0
intr = ^ c; quit = ^ \; erase = ^ ?; kill = ^ u;
eof = ^ d; eol = <undef>; eol2 = <undef>; swtch = <undef>;
start = ^ q; stop = ^ s; susp = ^ z; dsusp = ^ y;
rprnt = ^ r; flush = ^ o; werase = ^ w; lnext = ^ v;
-parenb -parodd cs8 -cstopb -hupcl cread -clocal -loblk -crtscts -crtsxoff
-parext
-ignbrk brkint ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl -iuclc
ixon -ixany -ixoff imaxbel
isig icanon -xcase echo echoe echok -echonl -noflsh
-tostop echoctl -echoprt echoke -defecho -flusho -pendin iexten
opost -olcuc onlcr -ocrnl -onocr -onlret -ofill -ofdel tab3
                                                       ^^^^
# &#12356;&#12288;&#12288;&#12288;&#12288;&#12288; &#8592; "&#12354;&#12356;"
is entered
ksh: i: not found [No such file or directory]

2. set -o o/p below(which is the same across both rlogin and telnet/ssh)

--- the Command output result of "set -o" when connecting to rlogin/telnet

(set-o_rlogin.txt, set-o_telnet.txt)
Current option settings
allexport                off
bgnice                   on
braceexpand              on
clobber                  on
emacs                    off
errexit                  off
exec                     on
glob                     on
globstar                 on
gmacs                    on
histexpand               off
ignoreeof                off
interactive              on
keyword                  off
letoctal                 off
log                      on
login_shell              off
markdirs                 off
monitor                  on
multiline                on
notify                   off
pipefail                 off
privileged               off
rc                       on
restricted               off
showme                   off
trackall                 off
unset                    on
verbose                  off
vi                       off
viraw                    on
xtrace                   off

[rphaniram]

Any inputs from the above data you had requested ? Do you still believe this to be a rlogin issue ? If so, could you explain why for with same environment settings and stty setting as for a ssh/telnet session, we still see the issue on rlogin ?

[krader1961]

@rphaniram I cannot reproduce your problem. I should be able to do so given our current knowledge of the problem. However, I am not using rlogin since I do not have it enabled on any of my systems. I cannot reproduce this using a shell launched directly by my terminal emulator or over a ssh session.

This might be a bug in ksh. However, since no one else can reproduce the problem, at least without using rlogin, the simplest explanation is this problem is due to your environment and is not due to a bug in ksh. The onus is on you to provide a way to reproduce the problem that does not involve rlogin. Alternatively explain why this only affects rlogin sessions since their behavior should be identical to an interactive ssh session.

[krader1961]

Also, @rphaniram, please note that problems involving Unicode code points that are outside of the ASCII range of code points is quite common. There are many programs that do not correctly handle UTF-8 encoded sequences of code points. Whether or not the locale env vars are set correctly.

[rphaniram]

Thanks for the response Kurtis ! I shall replay the response to my CU who is actually hitting the problem.

If there more data that I could share on the points that you make, shall post to the thread.