salt-cp - UnicodeEncodeError in files with non-ascii characters

[dkfsalt]

Description of Issue

Using salt-cp to copy a file with non-ascii characters causes a decode error (and an incomplete copy).

Setup

Salt: 2019.2.0

Steps to Reproduce Issue

Setting up for the test...

# mkdir -p /root/test
# cd /root/test
# rm -f /root/test/*
# curl -s https://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html -o utf8.txt

# salt ip-10-0-253-180-centos1 test.ping
ip-10-0-253-180-centos1:
    True

Ensure environment is sane...

# systemctl restart salt-api salt-master
# salt ip-10-0-253-180-centos1 test.ping
ip-10-0-253-180-centos1:
    True

Reproduce issue....

# salt-cp ip-10-0-253-180-centos1 utf8.txt /tmp/

ip-10-0-253-180-centos1:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1607, in _thread_return
        return_data = minion_instance.executors[fname](opts, data, func, args, kwargs)
      File "/usr/lib/python2.7/site-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/salt/modules/cp.py", line 84, in recv
        fp_.write(data)
    UnicodeEncodeError: 'ascii' codec can't encode characters in position 347-382: ordinal not in range(128)

And from /var/log/salt/master...

2019-06-14 15:29:24,503 [salt.master      :1577][ERROR   ][15675] Received minion error from [ip-10-0-253-180-centos1]: The minion function caused an exception

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

# salt-master --versions
/usr/lib/python2.7/site-packages/salt/scripts.py:102: DeprecationWarning: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date.  Salt will drop support for Python 2.7 in the Sodium release or later.
Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.6.0
       cherrypy: 5.6.0
       dateutil: 2.8.0
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.26.3
        libnacl: Not Installed
       M2Crypto: 0.31.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.4
         Python: 2.7.5 (default, Apr  9 2019, 14:30:50)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.1.3.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

[dkfsalt]

Workaround

Use the --chunked feature to send any/all files. This seems to avoid the issue with the non-ascii characters.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[sagetherage]

@dkfsalt is this still an issue in later versions of Salt, I assume so, just checking to be sure.

[dfidler]

Yes, on a 3002. master/minion.

I'd move the severity to low though as there is a workaround mentioned in https://github.com/saltstack/salt/issues/53490#issuecomment-502193818

[Whissi]

I run into the same problem on Debian 11 (3002.6). Steps to reproduce:

# wget -O /tmp/testfile.dat "https://github.com/saltstack/salt/releases/download/v3005rc2/salt-ssh-3005rc2.tar.gz"
# salt-cp '*' /tmp/testfile.dat /tmp/
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
Traceback (most recent call last):
  File "/usr/bin/salt-cp", line 33, in <module>
    sys.exit(load_entry_point('salt==3002.6', 'console_scripts', 'salt-cp')())
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 429, in salt_cp
    client.run()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 53, in run
    cp_.run()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 145, in run
    ret = self.run_oldstyle()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 153, in run_oldstyle
    arg = [self._load_files(), self.opts["dest"]]
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 128, in _load_files
    files.update(self._file_dict(fn_))
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 117, in _file_dict
    data = fp_.read()
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
Traceback (most recent call last):
  File "/usr/bin/salt-cp", line 33, in <module>
    sys.exit(load_entry_point('salt==3002.6', 'console_scripts', 'salt-cp')())
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 429, in salt_cp
    client.run()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 53, in run
    cp_.run()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 145, in run
    ret = self.run_oldstyle()
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 153, in run_oldstyle
    arg = [self._load_files(), self.opts["dest"]]
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 128, in _load_files
    files.update(self._file_dict(fn_))
  File "/usr/lib/python3/dist-packages/salt/cli/cp.py", line 117, in _file_dict
    data = fp_.read()
  File "/usr/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

Using "--chunked" works but doesn't help when not all minions supporting 'cp.recv_chunked' yet.