utf-8 and Windows

[vglavnyy]

I need to log utf-8 string, without "u8-to-wide-to-narrow" conversions under Windows. LOG_DEBUG << u8"мир-пир"; What is the simplest way for this under Windows?

[SergiusTheBest]

I'll take a look at what can be done.

[danieldietsch]

Is there any progress on this -- or some plans?

[SergiusTheBest]

@danieldietsch Just to clarify: do you use char8_t or char?

[danieldietsch]

I basically threat the source files under windows as utf-8 with msvc and /utf-8. It would be nice if I could do

PLOGN << "😀🦀";

and always get 😀🦀 in a Windows terminal.

[danieldietsch]

It seems that the unicode support for Windows is somewhat buggy. I use msvc with /utf8 and plog::ColorConsoleAppender and try the following.

    PLOGN << u8"u8😀🦀";
    PLOGN << u"u😀🦀";
    PLOGN << U"U😀🦀";
    PLOGN << L"L😀🦀";
    PLOGN << "😀🦀";

Windows Terminal, cmd, chcp 437 (same with bash) conhost, cmd, chcp 437 (somewhat expected because the font does not support unicode)

Now the same written to a file and displayed with vscode Windows Terminal, cmd, chcp 437 (same with bash) conhost, cmd, chcp 437

Is there a way to write unicode characters consistently right now?

Background colors seem also broken in Windows Terminal: conhost looks ok:

[SergiusTheBest]

@danieldietsch The Windows way is to use wchar_t everywhere. How did write a log to the file?

[danieldietsch]

Just redirecting the output CXX11.exe > out.log

Using wchar_t everywhere does not make sense for multiplatform projects (or at least for us it does not).

If we could simply choose which codePage is passed to toNarrow and used in toWide we could switch that to kUTF8 and use /utf8 during compilation; then even with Windows it would pass "😀🦀" as expected (*):

static plog::ColorConsoleAppender<plog::TxtFormatter> appender{};
    plog::init(plog::debug, &appender);


    PLOG_FATAL << "Fatal";
    PLOG_INFO << "Info";

    PLOGD << u8"u8😀🦀";
    PLOGD << u"u😀🦀";
    PLOGD << U"U😀🦀";
    PLOGD << L"L😀🦀";
    PLOGD << "😀🦀";
    PLOGD << PLOG_NSTR("PLOG_NSTR 😀🦀");

But I just saw, a better way is probably https://github.com/SergiusTheBest/plog/issues/239

(*) except for the background color; for that one would probably need to switch to https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences and use the SetConsoleMode API with the ENABLE_VIRTUAL_TERMINAL_PROCESSING (0x0004) flag. But I am not sure.

[SergiusTheBest]

@danieldietsch I'm adding support for UTF8 everywhere. It will automatically detect /utf8.

[danieldietsch]

Cool, looking forward to it!

[SergiusTheBest]

@danieldietsch @vglavnyy Support for Utf8Everywhere and char8_t is merged to master.