No mapping for the Unicode character exists in the target multi-byte code page

[ItzLevvie]

Brief description of your issue

Logs from the DiagOutputDir folder with --verbose-logs:

1. winget validate = https://pastebin.com/raw/a3WAmzPN
2. winget install = https://pastebin.com/raw/A3xFjEwQ
3. winget source update = https://pastebin.com/raw/GBp1un8Q

My settings.json file = https://pastebin.com/raw/q68y6PLX

Steps to reproduce

Install TTKN.CNKIExpress version 0.1.20 from https://github.com/microsoft/winget-pkgs/pull/56247.

PR files: https://github.com/vedantmgoyal2009/winget-pkgs/tree/b00cfb3c7534916f0c75b962235215caf185afc4/manifests/t/TTKN/CNKIExpress/0.1.20

Expected behavior

For the TTKN.CNKIExpress WinGet package to install successfully on my PC or VM.

Actual behavior

I'm seeing the error below on Windows Terminal, Command Prompt, and Windows PowerShell:

An unexpected error occurred while executing the command:
No mapping for the Unicode character exists in the target multi-byte code page.

which prevents me from installing the TTKN.CNKIExpress WinGet package.

Environment

Windows Package Manager (Preview) v1.3.1661-preview
Copyright (c) Microsoft Corporation. All rights reserved.

Windows: Windows.Desktop v10.0.25140.1000
System Architecture: X64
Package: Microsoft.DesktopAppInstaller v1.18.1661.0

Logs: %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\DiagOutputDir

Links
---------------------------------------------------------------------------
Privacy Statement   https://aka.ms/winget-privacy
License Agreement   https://aka.ms/winget-license
Third Party Notices https://aka.ms/winget-3rdPartyNotice
Homepage            https://aka.ms/winget
Windows Store Terms https://www.microsoft.com/en-us/storedocs/terms-of-sale

Any other software?

1. Windows Terminal Preview version 1.14.1452.0
2. Windows Package Manager Source (winget) version 2022.620.1418.424
3. Git version 2.37.0.rc1.windows.1

[ItzLevvie]

I was also able to repro this on https://github.com/microsoft/winget-pkgs/pull/56499 as well.

Looks like it's likely the Chinese characters in the InstallerUrl that's causing this issue because if I change the InstallerUrl to https://www.google.com it doesn't spew out the No mapping for the Unicode character exists in the target multi-byte code page. error.

[ItzLevvie]

Azure Pipelines in the winget-pkgs repository is now hitting this error: https://github.com/microsoft/winget-pkgs/pull/59030#issuecomment-1113503092

[florelis]

It looks like this is triggered by having Unicode characters in the URL, and the culprit is this line: https://github.com/microsoft/winget-cli/blob/5390e1c7873bf9455261be189083498bcb2e8dc1/src/AppInstallerCLICore/Workflows/DownloadFlow.cpp#L72 Specifically, this part: filename.string()

Looks like changing it to filename.u8string() fixes it. We could send a quick PR to fix that, but maybe we'd want to spend some more time to think about side effects or adding some tests...

[Trenly]

@denelon - Can we maybe think about pulling this into 1.4? Based on @lechacon's comment above, it seems like it wouldn't be too difficult, and this bug is causing several packages to fail both validation and install for users

[denelon]

Yes, I've already asked engineering to look into this one since it appears to be a relatively small fix. Thanks for linking the PRs! This helps tremendously.

[vedantmgoyal9]

• https://github.com/microsoft/winget-pkgs/pull/75392
• https://github.com/microsoft/winget-pkgs/pull/70086
• https://github.com/microsoft/winget-pkgs/pull/64058
• https://github.com/microsoft/winget-pkgs/pull/62491
• https://github.com/microsoft/winget-pkgs/pull/70066
• https://github.com/microsoft/winget-pkgs/pull/62517

Sorry if any of them have been mentioned earlier.

[JohnMcPMS]

Looks like changing it to filename.u8string() fixes it. We could send a quick PR to fix that, but maybe we'd want to spend some more time to think about side effects or adding some tests...

It should be filename.wstring(), because we are trying to measure the number of characters in the final name, and on Windows that means in UTF-16 (WCS2) characters. I failed when I wrote it, although admittedly I believed before checking it just now that string might be returning the internal representation.