[Noetic] UnicodeEncodeError: 'charmap' codec can't encode character

[seanyen]

Describe the bug

When you run catkin_make, you may encounter the errors and a similar traceback like below:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f916' in position 2385: character maps to <undefined>
Traceback (most recent call last):
  File "D:/a/1/s/samples/warehouse/ros_ws/build/catkin_generated/generate_cached_setup.py", line 22, in <module>
    code = generate_environment_script('D:/a/1/s/samples/warehouse/ros_ws/devel/env.bat')
  File "C:\opt\ros\noetic\x64\lib\site-packages\catkin\environment_cache.py", line 62, in generate_environment_script
    output = subprocess.check_output([env_script, sys.executable, '-c', python_code])
  File "subprocess.py", line 411, in check_output
  File "subprocess.py", line 512, in run
subprocess.CalledProcessError: Command '['D:/a/1/s/samples/warehouse/ros_ws/devel/env.bat', 'C:\\opt\\ros\\noetic\\x64\\python.exe', '-c', 'import os; print(dict(os.environ))']' returned non-zero exit status 1.
CMake Error at C:/opt/ros/noetic/x64/share/catkin/cmake/safe_execute_process.cmake:11 (message):
  execute_process(C:/opt/ros/noetic/x64/python.exe
  "D:/a/1/s/samples/warehouse/ros_ws/build/catkin_generated/generate_cached_setup.py")
  returned error code 1
Call Stack (most recent call first):
  C:/opt/ros/noetic/x64/share/catkin/cmake/all.cmake:208 (safe_execute_process)
  C:/opt/ros/noetic/x64/share/catkin/cmake/catkinConfig.cmake:20 (include)
  CMakeLists.txt:58 (find_package)

One possible cause is that some values in your environment variables are not able to be decoded by the current system codepage.

Before this issue is addressed in catkin, one way to workaround that is to set PYTOHNUTF8=1 and it asks Python to use UTF-8 to decode instead of using the system codepage.

[csorvagep]

I have a similar issue with the UnicodeDecodeError: 'utf-8' codec can't decode ... error on the noetic distro.

I think this is the other way around, my variable is not coded with UTF-8 and python cannot decode it. The set PYTHONUTF8=1 did not help.

[ooeygui]

@csorvagep Thank you for the report. Can you provide more details on the problem so we can reproduce it?

[csorvagep]

I have asked a question here first: https://answers.ros.org/question/369594/ros-on-windows-noetic-failed-to-build-an-empty-workspace-utf8-decode-error/ Buf after the comment, I've also created an issue here: https://github.com/ros/catkin/issues/1131

Here's more detail: I've installed ROS noetic on Windows 10 20H2 with MSVC v14.28, as in the installation tutorial. After creating an empty workspace with catkin_init_workspace e.g.: C:\catkin_ws, the catkin_make command fails with the following:

-- catkin 0.8.9
-- BUILD_SHARED_LIBS is on
Traceback (most recent call last):
  File "C:/catkin_ws/build/catkin_generated/generate_cached_setup.py", line 22, in <module>
    code = generate_environment_script('C:/catkin_ws/devel/env.bat')
  File "C:\opt\ros\noetic\x64\lib\site-packages\catkin\environment_cache.py", line 63, in generate_environment_script
    env_after = ast.literal_eval(output.decode('utf8'))
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfb in position 2928: invalid start byte
CMake Error at C:/opt/ros/noetic/x64/share/catkin/cmake/safe_execute_process.cmake:11 (message):
  execute_process(C:/opt/ros/noetic/x64/python.exe
  "C:/catkin_ws/build/catkin_generated/generate_cached_setup.py") returned
  error code 1
Call Stack (most recent call first):
  C:/opt/ros/noetic/x64/share/catkin/cmake/all.cmake:208 (safe_execute_process)
  C:/opt/ros/noetic/x64/share/catkin/cmake/catkinConfig.cmake:20 (include)
  CMakeLists.txt:58 (find_package)

Turns out, I have OneDrive set up with my company account, which creates environment variables for its directory. The directory name includes the name of the company which conatins the character 'ű' (u with double acute), which causes this error. I don't know what is the current character encoding on my system, but probably latin2 or similar.

To replicate this, just create a user environment variable e.g.: foo=ű (I've only tried with the gui), then open a new ROS termianal, and call catkin_make on a clean workspace.

As a workaround I've created a small batch file, which unsets these variables, and the build succeeds.

[leader1313]

@csorvagep exactly same as my bug of catkin make on windows ROS, I disconnect the "Onedrive" account then it works without any errors.

[csorvagep]

In case you need the Onedrive account, it is enough to clear the variables on the current terminal. Because the windows ROS needs other initializations as well I used this batch file:

@echo off
call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\Tools\VsDevCmd.bat" -arch=amd64 -host_arch=amd64
set ChocolateyInstall=c:\opt\chocolatey
call c:\opt\ros\noetic\x64\setup.bat
set OneDrive=''
set OneDriveCommercial=''
cd C:\catkin_ws

[e71828]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef in position 794: invalid continuation byte

totally the same error, however, because of my computer's name is 風の住む町.