steamctl icon indicating copy to clipboard operation
steamctl copied to clipboard
verified publisher icon ValvePython

[BUG] UnicodeEncodeError: 'charmap' codec can't encode character

Open woctezuma opened this issue 3 years ago • 4 comments

Description I want to export the list of apps to a text file using the Windows console, and there is an UnicodeEncodeError error.

Steps to Reproduce the behavior

Run: steamctl apps list > apps_list.txt

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Wok\.local\bin\steamctl.exe\__main__.py", line 7, in <module>
  File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\__main__.py", line 59, in main
    rcode = cmd_func(args=args)
            ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\commands\apps\gcmds.py", line 97, in cmd_apps_list
    print(app_id, app_names.get(app_id, 'Unknown App {}'.format(app_id)))
  File "C:\Users\Wok\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u221e' in position 11: character maps to <undefined>

Expected behavior The software should run without any error, and the text file should contain the lists of appIDs along the app names.

What actually happend

$ head apps_list.txt

5 Dedicated Server
7 Steam Client
8 winui2
10 Counter-Strike
20 Team Fortress Classic
30 Day of Defeat
40 Deathmatch Classic
50 Half-Life: Opposing Force
60 Ricochet
70 Half-Life

$ tail apps_list.txt

208090 Unknown App 208090
208140 ENDLESSâ–’ Space - Definitive Edition
208157 ENDLESSâ–’ Space - Amplitude Badge + GAMES2GETHER Points
208286 Train Simulator: Sherman Hill Route Add-On
208540 Dungeon Defenders Lucky Community Costume Pack
208546 Dungeon Denders - Jester Hero
208570 Dishonored: Dunwall City Trials DLC
208575 Dishonored: The Knife of Dunwall
208580 STAR WARSâ–’ Knights of the Old Republicâ–’ II: The Sith Lordsâ–’
208610

The error happens for appID=208610 which is Skullgirls ∞Endless Beta∞. I believe the culprit is the ∞ character.

Logs

steamctl -l debug 
[DEBUG] steamctl: Parsed args: {'versions_report': None, 'log_level': 'debug', 'anonymous': False, 'user': None, 'password': None, 'command': 'apps', '_cmd_func': 'steamctl.commands.apps.gcmds:cmd_apps_list', 'subcommand': 'list', 'all': False}
[DEBUG] SteamClient: Reading CM servers from 'C:\\Users\\Wok\\AppData\\Local\\steamctl\\steamctl\\client\\cm_servers.json'
[DEBUG] CMServerList: Added 80 new CM addresses.
[DEBUG] steamctl.utils.storage: Opening file (r): C:\Users\Wok\AppData\Local\steamctl\steamctl\client\lastuser
[INFO] SteamClient: Reusing previous username: XXX
[INFO] SteamClient: Hint: use 'steamctl --user <username> ...' to change
[DEBUG] steamctl.utils.storage: Opening file (r): C:\Users\Wok\AppData\Local\steamctl\steamctl\client\lastuser
[INFO] SteamClient: Attempting login with remembered credentials
[DEBUG] steamctl.utils.storage: Opening file (r): C:\Users\Wok\AppData\Local\steamctl\steamctl\client\XXX.key
[DEBUG] SteamClient: Attempting login
[DEBUG] SteamClient: Connect initiated.
[DEBUG] Connection: Attempting connection to ('162.254.192.75', 27017)
[DEBUG] Connection: Connected.
[DEBUG] SteamClient: Emit event: 'connected'
[DEBUG] SteamClient: Incoming: <Msg(<EMsg.ChannelEncryptRequest: 1303> | ChannelEncryptRequest)>
[DEBUG] SteamClient: Emit event: <EMsg.ChannelEncryptRequest: 1303>
[DEBUG] SteamClient: Securing channel
[DEBUG] SteamClient: Outgoing: <Msg(<EMsg.ChannelEncryptResponse: 1304> | ChannelEncryptResponse)>
[DEBUG] SteamClient: Incoming: <Msg(<EMsg.ChannelEncryptResult: 1305> | ChannelEncryptResult)>
[DEBUG] SteamClient: Emit event: <EMsg.ChannelEncryptResult: 1305>
[DEBUG] SteamClient: Channel secured
[DEBUG] SteamClient: Emit event: 'channel_secured'
[DEBUG] SteamClient: Outgoing: <MsgProto(<EMsg.ClientLogon: 5514> | CMsgClientLogon)>
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.Multi: 1> | CMsgMulti)>
[DEBUG] SteamClient: Emit event: <EMsg.Multi: 1>
[DEBUG] SteamClient: Multi: Unpacking
[DEBUG] SteamClient: Multi: Decompressing payload (655 -> 1207)
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.ClientServersAvailable: 5501> | not parsed)>
[DEBUG] SteamClient: Emit event: <EMsg.ClientServersAvailable: 5501>
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.ClientServersAvailable: 5501> | not parsed)>
[...]
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.Multi: 1> | CMsgMulti)>
[DEBUG] SteamClient: Emit event: <EMsg.Multi: 1>
[DEBUG] SteamClient: Multi: Unpacking
[DEBUG] SteamClient: Multi: Decompressing payload (20905 -> 68577)
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.ClientPICSProductInfoResponse: 8904> | not parsed)>
[DEBUG] SteamClient: Emit event: <EMsg.ClientPICSProductInfoResponse: 8904>
[DEBUG] SteamClient: Emit event: 'job_4'
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.Multi: 1> | CMsgMulti)>
[DEBUG] SteamClient: Emit event: <EMsg.Multi: 1>
[DEBUG] SteamClient: Multi: Unpacking
[DEBUG] SteamClient: Multi: Decompressing payload (24145 -> 72215)
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.ClientPICSProductInfoResponse: 8904> | not parsed)>
[DEBUG] SteamClient: Emit event: <EMsg.ClientPICSProductInfoResponse: 8904>
[DEBUG] SteamClient: Emit event: 'job_4'
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.Multi: 1> | CMsgMulti)>
[DEBUG] SteamClient: Emit event: <EMsg.Multi: 1>
[DEBUG] SteamClient: Multi: Unpacking
[DEBUG] SteamClient: Multi: Decompressing payload (3153 -> 9775)
[DEBUG] SteamClient: Incoming: <MsgProto(<EMsg.ClientPICSProductInfoResponse: 8904> | not parsed)>
[DEBUG] SteamClient: Emit event: <EMsg.ClientPICSProductInfoResponse: 8904>
[DEBUG] SteamClient: Emit event: 'job_4'
[DEBUG] Connection: Disconnected.
[DEBUG] SteamClient: Emit event: 'disconnected'
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Wok\.local\bin\steamctl.exe\__main__.py", line 7, in <module>
  File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\__main__.py", line 59, in main
    rcode = cmd_func(args=args)
            ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Wok\.local\pipx\venvs\steamctl\Lib\site-packages\steamctl\commands\apps\gcmds.py", line 97, in cmd_apps_list
    print(app_id, app_names.get(app_id, 'Unknown App {}'.format(app_id)))
  File "C:\Users\Wok\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u221e' in position 11: character maps to <undefined>

Versions Report

steamctl --versions-report 
steamctl: 0.9.5

Dependencies:
               steam: 1.4.3
             appdirs: 1.4.4
         argcomplete: 2.0.0
                tqdm: 4.64.1
               arrow: 1.2.3
            pyqrcode: 1.2.1
      beautifulsoup4: 4.11.1
                 vpk: 1.4.0
                 vdf: 3.4
 gevent-eventemitter: 2.1
              gevent: 22.10.2
            greenlet: 2.0.1
              pyyaml: Not Installed
       pycryptodomex: 3.15.0
            protobuf: 3.20.3

Python runtime:
          executable: C:\Users\Wok\.local\pipx\venvs\steamctl\Scripts\python.exe
             version: 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)]
            platform: win32

System info:
              system: Windows
             machine: AMD64
             release: 10
             version: 10.0.19044

woctezuma avatar Nov 19 '22 23:11 woctezuma

There is no error if I run:

 steamctl apps list

I can even get the ∞ character to be rendered correctly:

208570 Dishonored: Dunwall City Trials DLC
208575 Dishonored: The Knife of Dunwall
208580 STAR WARSâ„¢ Knights of the Old Republicâ„¢ II: The Sith Lordsâ„¢
208610 Skullgirls ∞Endless Beta∞
208620 Midnight Mysteries: Salem Witch Trials
208630 Midnight Mysteries 3: Devil on the Mississippi
208640 Midnight Mysteries 4: Haunted Houdini

However, the list is really long, so it gets truncated by the Windows console, and it would be handier to save it to a text file.

I remember encountering this issue in the past, without reporting it, and then editing the code to save the output to a file. I wonder if there is a better fix.

On a side-note, I am not sure if the issue only happens on Windows.

woctezuma avatar Nov 19 '22 23:11 woctezuma

The error can be reproduced with a Python file scratch_1.py, unrelated to steamctl:

print('∞')

python scratch_1.py

Output:

∞

python scratch_1.py > temp.txt

Output:

Traceback (most recent call last):
  File "C:\Users\Wok\AppData\Roaming\JetBrains\PyCharmCE2022.2\scratches\scratch_1.py", line 1, in <module>
    print('∞')
  File "C:\Users\Wok\AppData\Local\Programs\Python\Python311\Lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeEncodeError: 'charmap' codec can't encode character '\u221e' in position 0: character maps to <undefined>

woctezuma avatar Nov 20 '22 14:11 woctezuma

Use a terminal that supports utf8 and is configured to use utf8

rossengeorgiev avatar Nov 20 '22 15:11 rossengeorgiev

Use a terminal that supports utf8 and is configured to use utf8

That is the case. The issue is with the redirection to a file, which does not seem to work properly on Windows.

For instance, in the picture below, there is no error, but an inconsistency.

echo ∞
echo ∞ > temp.txt
type temp.txt

Windows terminal: echo ∞

I can fix the inconsistency with chcp 65001, but that does not fix the error when redirecting the output of a Python script.

chcp 65001
echo ∞
echo ∞ > temp.txt
type temp.txt

Windows terminal: chcp 65001

For the error with Python, I have a small workaround.

woctezuma avatar Nov 20 '22 15:11 woctezuma