pyinfra icon indicating copy to clipboard operation
pyinfra copied to clipboard
verified publisher icon pyinfra-dev

sftp operations fail if login script echos to stdout

Open znd4 opened this issue 4 years ago • 2 comments

Describe the bug

Whenever a host.put_file sftp operation gets called, if the target server's login server prints anything to stdout, paramiko will fail and throw a rather unhelpful SFTPError("Garbage packet received"), and the deploy script fails.

To Reproduce

  1. include a line like echo "hello world" in the .bashrc of a remote testing machine.
  2. Run an operation against that machine that requires a sudo password (or instantiate a host and host.put_file(io.StringIO("foo"), "bar.txt"))

Expected behavior

File would successfully be copied over and script continues executing

Meta

  • Include output of pyinfra --support.
--> Support information:

    If you are having issues with pyinfra or wish to make feature requests, please
    check out the GitHub issues at https://github.com/Fizzadar/pyinfra/issues .
    When adding an issue, be sure to include the following:

    System: Linux
      Platform: Linux-5.13.0-22-generic-x86_64-with-glibc2.34
      Release: 5.13.0-22-generic
      Machine: x86_64
    pyinfra: v1.5
    Executable: /home/zane/.local/bin/pyinfra
    Python: 3.10.0 (CPython, GCC 11.2.0)

  • How was pyinfra installed (source/pip)? pipx

  • Include pyinfra-debug.log (if one was created)

  • Consider including output with -vv and --debug.

at 15:08:54 🔨 ❯ eval $(ssh-agent) && cat ~/.ssh/id_rsa | ssh-add - && USE_SUDO_PASSWORD=$(read -s -p "password: ") pyinfra
--user=zane --use-sudo-password --port=$SSH_PORT $IP ~/.dotfiles/install.py --debug -vv --sudo-user=zane
Agent pid 36226
Identity added: (stdin) ((stdin))
password:     [pyinfra_cli.main] Checking potential directory: /home/zane/.dotfiles
    [pyinfra_cli.main] Deploy directory remains as cwd
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...

--> Connecting to hosts...
    [pyinfra.api.connectors.ssh] Connecting to: 172.17.0.1 ({'allow_agent': True, 'look_for_keys': True, 'hostname': '172.17
.0.1', '_pyinfra_force_forward_agent': None, 'username': 'zane', 'port': 2222, 'timeout': 10})
    No host key for [172.17.0.1]:2222 found in known_hosts
    [172.17.0.1] Connected
    [pyinfra.api.state] Activating host: 172.17.0.1

--> Preparing operations...
    Loading: /home/zane/.dotfiles/install.py
    [pyinfra.api.operation] Adding operation, {'Add appimagelauncher repo'}, opOrder=(381, 13, 24), opHash=321afa3d93fafcc70
5064a4cf653489d135ba55b
    [pyinfra.api.operation] Adding operation, {'Add signal desktop key'}, opOrder=(381, 13, 29), opHash=0c4a5be615eee7a75aa7
2729c583af3ab91f8a76
    [pyinfra.api.facts] Getting fact: apt_keys () (ensure_hosts: (Host(172.17.0.1),))
> /home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/util.py(170)_get_sudo_password()
-> host.put_file(get_sudo_askpass_exe(), SUDO_ASKPASS_EXE_FILENAME)
(Pdb) c
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/host.py", line 200, in run_shell_comma
nd
    return self.executor.run_shell_command(self.state, self, *args, **kwargs)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/ssh.py", line 317, in run_s
hell_command
    return_code, combined_output = execute_command_with_sudo_retry(
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/util.py", line 55, in execu
te_command_with_sudo_retry
    return_code, combined_output = execute_command()
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/ssh.py", line 284, in execu
te_command
    unix_command = make_unix_command_for_host(state, host, command, **command_kwargs)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/util.py", line 235, in make
_unix_command_for_host
    command_kwargs['sudo_password'] = _get_sudo_password(host, use_sudo_password)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/util.py", line 170, in _get
_sudo_password
    host.put_file(get_sudo_askpass_exe(), SUDO_ASKPASS_EXE_FILENAME)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/host.py", line 204, in put_file
    return self.executor.put_file(self.state, self, *args, **kwargs)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/ssh.py", line 487, in put_f
ile
    _put_file(host, filename_or_io, remote_filename)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/ssh.py", line 415, in _put_
file
    sftp = _get_sftp_connection(host)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/util.py", line 66, in wrapper
    value = func(*args, **kwargs)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/ssh.py", line 338, in _get_
sftp_connection
    return SFTPClient.from_transport(transport)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/paramiko/sftp_client.py", line 170, in from_transp
ort
    return cls(chan)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/paramiko/sftp_client.py", line 130, in __init__
    server_version = self._send_version()
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/paramiko/sftp.py", line 134, in _send_version
    t, data = self._read_packet()
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/paramiko/sftp.py", line 205, in _read_packet
    raise SFTPError("Garbage packet received")
paramiko.sftp.SFTPError: Garbage packet received
2021-12-25T20:09:08Z <Greenlet at 0x7fd30aeab880: <bound method Host.run_shell_command of Host(172.17.0.1)>(StringCommand(!
command -v apt-key >/dev/null || !, sudo=True, sudo_user=None, use_sudo_password=True, su_user=None, timeout=None, env={}, s
hell_executable=None, print_output=False, print_input=True, return_combined_output=True)> failed with SFTPError

--> An unexpected exception occurred in: /home/zane/.dotfiles/install.py:

  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra_cli/util.py", line 80, in exec_file
    exec(PYTHON_CODES[filename], data)
  File "/home/zane/.dotfiles/install.py", line 381, in <module>
    main()
  File "/home/zane/.dotfiles/install.py", line 13, in main
    configure_repos()
  File "/home/zane/.dotfiles/install.py", line 29, in configure_repos
    apt.key(
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/operation.py", line 365, in decorated_
func
    commands = unroll_generators(func(*actual_args, **actual_kwargs))
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/util.py", line 192, in unroll_generato
rs
    for item in generator:
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/operations/apt.py", line 69, in key
    existing_keys = host.get_fact(AptKeys)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/host.py", line 138, in get_fact
    return get_host_fact(self.state, self, name_or_cls, args=args, kwargs=kwargs)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/facts.py", line 334, in get_host_fact
    fact_data = get_facts(state, name, args=args, kwargs=kwargs, ensure_hosts=(host,))
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/facts.py", line 276, in get_facts
    status, combined_output_lines = greenlet.get()
  File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get
  File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/gevent/_compat.py", line 65, in reraise
    raise value.with_traceback(tb)
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/host.py", line 200, in run_shell_comma
nd
    return self.executor.run_shell_command(self.state, self, *args, **kwargs)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/ssh.py", line 317, in run_s
hell_command
    return_code, combined_output = execute_command_with_sudo_retry(
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/util.py", line 55, in execu
te_command_with_sudo_retry
    return_code, combined_output = execute_command()
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/ssh.py", line 284, in execu
te_command
    unix_command = make_unix_command_for_host(state, host, command, **command_kwargs)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/util.py", line 235, in make
_unix_command_for_host
    command_kwargs['sudo_password'] = _get_sudo_password(host, use_sudo_password)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/util.py", line 170, in _get
_sudo_password
    host.put_file(get_sudo_askpass_exe(), SUDO_ASKPASS_EXE_FILENAME)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/host.py", line 204, in put_file
    return self.executor.put_file(self.state, self, *args, **kwargs)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/ssh.py", line 487, in put_f
ile
    _put_file(host, filename_or_io, remote_filename)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/ssh.py", line 415, in _put_
file
    sftp = _get_sftp_connection(host)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/util.py", line 66, in wrapper
    value = func(*args, **kwargs)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/pyinfra/api/connectors/ssh.py", line 338, in _get_
sftp_connection
    return SFTPClient.from_transport(transport)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/paramiko/sftp_client.py", line 170, in from_transp
ort
    return cls(chan)
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/paramiko/sftp_client.py", line 130, in __init__
    server_version = self._send_version()
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/paramiko/sftp.py", line 134, in _send_version
    t, data = self._read_packet()
  File "/home/zane/.local/pipx/venvs/pyinfra/lib/python3.10/site-packages/paramiko/sftp.py", line 205, in _read_packet
    raise SFTPError("Garbage packet received")
paramiko.sftp.SFTPError: Garbage packet received

znd4 avatar Dec 25 '21 20:12 znd4

According to someone in the paramiko issue I linked, the scp library/plugin for paramiko doesn't have this drawback. I don't know enough about the differences between sftp and scp to know whether it'd make sense to change over (even ignoring the effort involved), but I thought it'd be worth calling out.

For now, I probably shouldn't have had my login scripts echoing to stdout anyways, so I've got some cleanup to do.

I can imagine this being a blocker for people who want to deploy to servers that they don't have full control over though

znd4 avatar Dec 25 '21 20:12 znd4

Hi @zdog234 thank you for raising this, definitely needs fixing I will investigate the scp library some more as that looks like a sensible solution. Pyinfra isn't using any of the other SFTP features so don't have any concerns with the swap 👍

Fizzadar avatar Jan 04 '22 08:01 Fizzadar