Add support for non-POSIX target environments in the SSH connector

[KuxaBeast]

Description

I tried using pyinfra to write a few operations to deploy my infrastructure consisting of MikroTik RouterBoard-based devices running RouterOS, but found out, that the SSH connector has a hardcoded sh -c wrapper function in its code for executing remote commands.

After some investigation I found the culprit is this: (api/connectors/ssh.py:282) command = make_unix_command(command, state=state, **command_kwargs)

I don't know if I am missing something and there is a way to execute remote commands without this wrapper (which can only support devices running POSIX shell), but it would be nice to take into consideration adding support for non-POSIX target environments or at least to have the option to turn this feature off.

To Reproduce

Try to execute a valid command over SSH on virtually any target which doesn't ship a POSIX-compliant shell.

Expected behavior

The command executes without any issue on the target device.

Meta

• pyinfra --support

System: Linux
      Platform: Linux-5.14.14-arch1-1-x86_64-with-glibc2.33
      Release: 5.14.14-arch1-1
      Machine: x86_64
    pyinfra: v1.4.17
    Executable: /home/user/.local/bin/pyinfra
    Python: 3.9.7 (CPython, GCC 11.1.0)

• How was pyinfra installed (source/pip)? pipx, since that's the only way that worked for me on Arch Linux
• Output with -vv and --debug

$ pyinfra -vv --debug yum.ospf exec -- '/file print'
    [pyinfra_cli.main] Deploy directory remains as cwd
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...

--> Connecting to hosts...
    [pyinfra.api.connectors.ssh] Connecting to: yum.ospf ({'allow_agent': True, 'look_for_keys': True, 'hostname': 'yum.ospf', 'timeout': 10})
    [yum.ospf] Connected
    [pyinfra.api.state] Activating host: yum.ospf
    [pyinfra.api.operation] Adding operation, {'Server/Shell'}, opOrder=(0,), opHash=784a97bf1955d5f7a2b9dd6c1e371e17b73c42bc

--> Proposed changes:
    Ungrouped:
    [yum.ospf]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Server/Shell (/file print)
    [pyinfra.api.operations] Starting operation Server/Shell on yum.ospf
    [pyinfra.api.connectors.ssh] Running command on yum.ospf: (pty=None) sh -c '/file print'
[yum.ospf] >>> sh -c '/file print'
[yum.ospf] syntax error (line 1 column 7)
    [pyinfra.api.connectors.ssh] Waiting for exit status...
    [pyinfra.api.connectors.ssh] Command exit status: 1
    [yum.ospf] Error
    [pyinfra.api.state] Failing hosts: yum.ospf
--> pyinfra error: No hosts remaining!

With the (api/connectors/ssh.py:282) line manually commented out:

$ pyinfra -vv --debug yum.ospf exec -- '/file print'
    [pyinfra_cli.main] Deploy directory remains as cwd
--> Loading config...
--> Loading inventory...
    [pyinfra_cli.inventory] Creating fake inventory...

--> Connecting to hosts...
    [pyinfra.api.connectors.ssh] Connecting to: yum.ospf ({'allow_agent': True, 'look_for_keys': True, 'hostname': 'yum.ospf', 'timeout': 10})
    [yum.ospf] Connected
    [pyinfra.api.state] Activating host: yum.ospf
    [pyinfra.api.operation] Adding operation, {'Server/Shell'}, opOrder=(0,), opHash=784a97bf1955d5f7a2b9dd6c1e371e17b73c42bc

--> Proposed changes:
    Ungrouped:
    [yum.ospf]   Operations: 1   Commands: 1   

--> Beginning operation run...
--> Starting operation: Server/Shell (/file print)
    [pyinfra.api.operations] Starting operation Server/Shell on yum.ospf
    [pyinfra.api.connectors.ssh] Running command on yum.ospf: (pty=None) /file print
[yum.ospf] >>> /file print
[yum.ospf]  # NAME                   TYPE                        SIZE CREATION-TIME       
[yum.ospf]  0 skins                  directory                        jan/01/1970 02:00:05
[yum.ospf]  1 auto-before-reset.b... backup                   16.4KiB jan/01/1970 02:00:06
[yum.ospf]  2 pub                    directory                        jan/02/1970 05:23:38
        ... (output trimmed) ...
[yum.ospf] 
    [pyinfra.api.connectors.ssh] Waiting for exit status...
    [pyinfra.api.connectors.ssh] Command exit status: 0
    [yum.ospf] Success

--> Results:
    Ungrouped:
    [yum.ospf]   Successful: 1   Errors: 0   Commands: 1/1   

[KuxaBeast]

Now realized it should've probably been a feature request but, well.

[Fizzadar]

Hi @KuxaBeast - it is currently possible to set a shell, but not disable it. There's two options for this:

• shell_executable global argument passed into operations (https://docs.pyinfra.com/en/1.x/deploys.html#global-arguments)
• The SHELL config variable

What's the target system you're working on, out of interest?

[KuxaBeast]

What's the target system you're working on, out of interest?

It's RouterOS, a proprietary Linux-based software from Mikrotik, which supports many architectures and devices, mainly their RouterBoards. They have their own implementation of SSH server and scripting command interface.

[KuxaBeast]

@Fizzadar, what would be the best approach to implement/fix this feature? This should be actually an opt-in functionality in the connector, or completely done by the user manually IMHO. Or maybe make a separate function from server.shell (something like server.subprocess) - something that does not allow for shell expressions, just plainly sends the string command to the target?

• shell_executable global argument passed into operations

also, I don't think it's the best idea to assume that any possible shell_executable will have the, currently hardcoded, -c argument for executing expressions - I don't know what is the CLI for Windows' cmd, but if it's a variable, it could be set to anything else

[Fizzadar]

@KuxaBeast apologies for the delay here. Agreed there's a few issues with the current handling. Currently Windows has it's own make command function. The -c argument is pretty standard across most POSIX shell implementations but I agree this shouldn't be completely hardcoded. I would propose immediately:

• Allow setting shell_executable=None which removes entirely the wrapping of commands under a shell executable

This should be a relatively simple change to the make_unix_command.

And longer term:

• Making the -c configurable somehow, with -c being the default
• Better handling shell executable between Windows + Unix systems