Control characters (\x01 .. \x1F) are changed

[cebaa]

Versions

ConEmu build: 200713 x64 OS version: Windows 10 x64 Used shell version (Far Manager, git-bash, cmd, powershell, cygwin, whatever): cygwin, powershell

Problem description

Control characters changed.

Steps to reproduce

For Cygwin, the third invocation is me copying the output of the first invocation and pasting it within echo just to show they changed.

Cygwin within ConEmu:

$ echo -e a\\001b
a☺b
$ echo -e a\\001b|xxd
00000000: 6101 620a                                a.b.
$ echo "a☺b"|xxd
00000000: 61e2 98ba 620a                           a...b.
$

Cygwin within cmd.exe:

$ echo -e a\\001b
ab
$ echo -e a\\001b|xxd
00000000: 6101 620a                                a.b.
$ echo "a^Ab"|xxd
00000000: 6101 620a                                a.b.
$

Powershell within ConEmu:

> echo "a$([char]1)b"
a☺b
>

Powershell within cmd.exe:

> echo "a${[char]1}b"
ab
>

Let me know if there's something I need to change in the settings, etc.

Actual results

Control characters seem to be changed from byte 01 to bytes e2 98 ba.

Expected results

Control characters should remain the same.

Additional files

[Maximus5]

ConEmu doesn't change output of the applications. Only ANSI sequences are processed and stripped. But yours 0x001 is not a part of ANSI.

I believe you could get the same experience in bare cmd.exe, try to play with chcp, environment, etc. I don't think ConEmu could be responsible for that. It's all in conhost.

[cebaa]

Thanks Maximus5! I can replicate in plain cmd as well.

Cmd from ConEmu (running "%windir%\syswow64\cmd.exe"):

> c:\windows\syswow64\chcp 65001
Active code page: 65001

> type soh.vbs
WScript.StdOut.WriteLine "a" + chr(1) + "b"


> cscript -b soh.vbs
a☺b

> echo "a☺b"
"a☺b"

>

For the last echo, I copied the output of cscript -b soh.vbs and pasted it between quotes in echo "".

Cmd when run via Win+R:

> c:\windows\syswow64\chcp 65001
Active code page: 65001

> type soh.vbs
WScript.StdOut.WriteLine "a" + chr(1) + "b"


> cscript -b soh.vbs
ab

> echo "a^Ab"
"ab"

>

Note that when ab is copied and pasted into the last echo, it is pasted as a^Ab, not as a☺b.

Assuming you have Win 10 x64, could you run the same test case on your end when you get a chance to see what results you get?

Also, let me know if you have other suggestions I can try.

[cebaa]

@Maximus5 Just checking if you can reproduce this or if you have any other suggestions I can try.

[Maximus5]

Well, firstly I misunderstood you issue. Indeed, ConEmu uses unicode glyphs to draw characters \x01..\x1F. With this fix copying to clipboard will use unchanged data.

Note: character \x00 is still replaces with space, otherwise copying strings to clipboard is impossible and useless.

[bitraid]

ConEmu uses unicode glyphs to draw characters \x01..\x1F.

@Maximus5 For bitmap fonts, a fall-back proportional font is used, which looks ugly and breaks monospace. I experience this on fresh Windows installs (no extra font linking), while trying to set the bitmap version of terminus font, but happens with any bitmap font, like Courier. To demonstrate, set a bitmap font and run in powershell:

1..6+14..26+28..31 | %{"$_ |$([char]$_)|"}

I also noticed that with terminus font, many glyphs are displayed as empty boxes, even though they exist (for example, the Far Manager box symbols).

[Maximus5]

@bitraid It's up to user how to configure Alternative font https://conemu.github.io/en/SettingsFonts.html

[Maximus5]

https://conemu.github.io/blog/2020/10/11/Build-201011.html

[bitraid]

@Maximus5 The Alternative font makes no difference. It is only used for the selected unicode ranges. Low ascii characters are dispayed with a different font (when the main font is bitmap).

[cebaa]

@Maximus5 Thanks for fixing!

Just one more question:

$ echo -e a\\001b
ab
$ echo a^Ab

So I printed the SOH character and it shows as unicode (cannot copy after this change, but it's displayed as a☺b). Now, when I copy that and paste (second echo above), it shows as a^Ab. Similarly in cygwin I can do Ctrl+V Ctrl+A and it will insert a SOH displayed as ^A, not as ☺.

Is there a way to make ConEmu not change ^A to ☺, so that copy / paste works in a consistent manner with the display? I.e. to make it display as ^A always, as I think it did this at some time in the past.

[cebaa]

@Maximus5 Just checking if you have any thoughts about my previous comment.

[cebaa]

@Maximus5 Just checking if you have any thoughts about my comment above.