UNREACHABLE! => {"changed": false, "msg": "EOF on stream; last 100 lines received: The system cannot find the path specified, "unreachable": true}

[jasminehung]

Host : Ansible version 2.9.4 (on CentOS7)+ Mitogen version 2.9, python version 2.7.5 Target : Windows 2012

Similar error message with #597 but specifying ansible_python_interpreter=/usr/bin/python3 or ansible_python_interpreter=/usr/bin/python2 doesn't work.

I'm trying to test ansible connection with my windows target, everything works fine but causing Windows server high CPU usage, so I installed mitogen and edited ansible.cfg. Tasks that did not connect with my Windows server run successfully, but getting this error when running win_ping :

fatal: [my_windows_host]: UNREACHABLE! => {"changed": false, "msg": "EOF on stream; last 100 lines received: The system cannot find the path specified, "unreachable": true}`

I've tried using ssh and winrm, also tried downgraded my ansible to 2.8.8, but still facing the same error. By removing the line strategy = mitogen in ansible.cfg, the playbook can run successfully again. Is it because Mitogen doesn't support Windows target? Or what else can I try? Thanks.

[s1113950]

Ansible python interpreter discovery was added in the latest master of Mitogen, can you give the latest commit a try?

[jasminehung]

Yes I've tried the latest commit, but got the same error message. Is it because Mitogen doesn't support Windows target? After switching my target to a linux server, it can work fine.

[s1113950]

Mitogen supports Windows 🤔 Would you be able to provide a simple Dockerfile of your environment for me to test with? https://hub.docker.com/_/microsoft-windows I see MS has some official Docker images available. Also if you have a minimally-reproducible playbook I could use that would help out as well! :)

[jasminehung]

Can simply reproduce the error message by runnung without a playbook: ansible my_windows_host -i hosts -m win_ping

With only a single line in the hosts file : my_windows_host ansible_ssh_user=administrator ansible_shell_type=cmd

Sorry but because the company's security reason, I couldn't upload the dockerfile through internet. I'll see what I can do to provide you more information...

[s1113950]

Ok. Is your base dockerfile something from here by chance? https://hub.docker.com/_/microsoft-windows I don't have a Windows machine to test against personally so I'd need a container to be able to debug. If you can't provide a dockerfile, could you provide a dump of -vvvv when running with Mitogen? That might give me some clue as well as to what the issue is 😄

[jasminehung]

Here is the dump of -vvvv when running with Mitogen log20200629.txt (My Windows host is a Windows server 2012 R2 64bit)

[s1113950]

Can you run this command from your machine?

ssh -o “LogLevel ERROR” -l administrator -i /userap/appd_ap/ansible/private_key -o “Compression yes” -o “ServerAliveInterval 30” -o “ServerAliveCountMax 10” -o “BatchMode yes” -o “StrictHostKeyChecking no” -o “UserKnownHostsFile /dev/null” -o “GlobalKnownHostsFile /dev/null” -C -o ControlMaster=auto -o ControlPersist=60s my_windows_host /usr/bin/python2 -c "'import codecs,os,sys;=codecs.decode;exec(((“eNqFkc1OwzAQhM/NU+S2tmq1TqD8RLIE6gFxQEgRogeoUFI7YDW1LSdNKE/PNkU0KQdu+2lnPaNxyhbCVhOnnSI08KztkS5ChML6NaFJMMJZbl1MOIs4p0dOWZ88bqMDr0pbKZL2wfdh0YcWAQ2rHdqXWY2um1CIEGTmW20gzIzslupTrbZ1lpeqW0+3lZ/m2kzdrv6wBjDn6EQ2Ft1ho3ylrXlJzpadrTKN9shwm949c1iK4dlBg1iS4YINcQxko2v7rkzira1v6iKaqXItVcPPk+v4ahZToAG+1XpdKxIxeLh/euScvxrADCsrsWoazMUb2ZctrVMGKwafA514lUkSXfL4gjL40g5fKpw46hYM2hz2/Rfux2DezYdOT9Ttf+q/KaNhyt/fiuk3EUyyMQ==”.encode(),“base64”),“zip”))’"

If you can, then it looks like a Windows error: https://github.com/microsoft/vscode-remote-release/issues/961 I googled ssh The system cannot find the path specified and there's a bunch of Windows-related results. Someone mentioned https://github.com/microsoft/vscode-remote-release/issues/961#issuecomment-511276255 as a possible fix?

[jasminehung]

When running this command, it returns : The system cannot find the path specified. When only run the first half of the command:ssh -o "LogLevel ERROR" -l administrator -i /userap/appd_ap/ansible/private_key -o "Compression yes" -o "ServerAliveInterval 30" -o "ServerAliveCountMax 10" -o "BatchMode yes" -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -o "GlobalKnownHostsFile /dev/null" -C -o ControlMaster=auto -o ControlPersist=60s my_windows_host it can successully enter my Windows host’s command interface, seems like there’s no problem with ssh?

If run ssh -o "LogLevel ERROR" -l administrator -i /userap/appd_ap/ansible/private_key -o "Compression yes" -o "ServerAliveInterval 30" -o "ServerAliveCountMax 10" -o "BatchMode yes" -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -o "GlobalKnownHostsFile /dev/null" -C -o ControlMaster=auto -o ControlPersist=60s df15tckmwin03 /usr/bin/python2 it starts to return : “The system cannot find the path specified” again.

I’ve check python path(/usr/bin/python2) and PYTHONPATH environment variable on my linux server, both exists.

[s1113950]

Does /usr/bin/python2 exist on your windows host? The python command is what's ran on the target system

[jasminehung]

After moving python to /usr/bin/python2 on my windows host, there's no more The system cannot find the path specified error. But now when executing the same command : ssh -o “LogLevel ERROR” -l administrator -i /userap/appd_ap/ansible/private_key -o “Compression yes” -o “ServerAliveInterval 30” -o “ServerAliveCountMax 10” -o “BatchMode yes” -o “StrictHostKeyChecking no” -o “UserKnownHostsFile /dev/null” -o “GlobalKnownHostsFile /dev/null” -C -o ControlMaster=auto -o ControlPersist=60s my_windows_host /usr/bin/python2 -c "'import codecs,os,sys;=codecs.decode;exec(((“......" there's a SyntaxError: invalid syntax on import. If directly run the same python command in cmd or PowerShell on my host, it can run without error. I'm still figuring out why this happens, guessing maybe similar with this question: https://stackoverflow.com/questions/2043453/executing-multi-line-statements-in-the-one-line-command-line/2043499#2043499 (but the solution didn't work in my case)

[s1113950]

I just noticed something! According to the code, this is what the codecs part is supposed to be:

return self.get_python_argv() + [
            '-c',
            'import codecs,os,sys;_=codecs.decode;'
            'exec(_(_("%s".encode(),"base64"),"zip"))' % (encoded.decode(),)
        ]

Note the _ in many places. However, in your example, /usr/bin/python2 -c "'import codecs,os,sys;=codecs.decode;exec(((“......" there are no _ 🤔

Would there be any reason as to why _ don't show up? The SyntaxError is likely because =codecs.decode; is not valid python, it should be _=codecs.decode

[jasminehung]

The _ did exist in my command but it disappear since I copy it to the txt file, sorry for the mistake. With _ in the command, the same error showed up on import (SyntaxError : invalid syntax).

[s1113950]

Ok. What Python2 version is on your target Windows host?

[jasminehung]

It's Python 2.7

[s1113950]

Ok. What's the minor version of it? Also I just noticed something else:

/usr/bin/python2 -c "'import codecs,os,sys;=codecs.decode;exec(((“......" there's a SyntaxError: invalid syntax on import.

from here, there's a weird quote in this part: codecs.decode;exec(((“......". I wonder if that has anything to do with it?

I would play with the syntax and try and run a simple command to see if it's an issue with what Mitogen creates:

ssh -o “LogLevel ERROR” -l administrator -i /userap/appd_ap/ansible/private_key -o “Compression yes” -o “ServerAliveInterval 30” -o “ServerAliveCountMax 10” -o “BatchMode yes” -o “StrictHostKeyChecking no” -o “UserKnownHostsFile /dev/null” -o “GlobalKnownHostsFile /dev/null” -C -o ControlMaster=auto -o ControlPersist=60s my_windows_host /usr/bin/python2 -c "'import sys;print sys.version_info'"

does something like that work? If it doesn't, I wonder if the extra quotes around the import statement are what's causing the issue when ran through ssh.

You could try

ssh -o “LogLevel ERROR” -l administrator -i /userap/appd_ap/ansible/private_key -o “Compression yes” -o “ServerAliveInterval 30” -o “ServerAliveCountMax 10” -o “BatchMode yes” -o “StrictHostKeyChecking no” -o “UserKnownHostsFile /dev/null” -o “GlobalKnownHostsFile /dev/null” -C -o ControlMaster=auto -o ControlPersist=60s my_windows_host /usr/bin/python2 -c "import sys;print sys.version_info"

that version if the previous version didn't work (removing the inner single quotes ')

I don't have any experience using cmd through ssh myself (I don't develop on Windows machines), and have no way of replicating the error on my end so I'm not sure how to help further 😞

[jasminehung]

Python 2.7.18 I’ve tried all the possible syntax around python command, but still Syntax error. I’ll keep on trying other possible code and if I have a solution I’ll leave a message here. Still great thanks for your helping!

[shinebayar-g]

My ansible is giving exact same error when I try to use mitogen. Remote server is Ubuntu 18.04.5 LTS and default python 3.6.9 is available on /usr/bin/python3

My ansible version: 2.9.15 Mitogen version: mitogen-0.2.9

$ ansible -i inventory app -m setup
my-server | UNREACHABLE! => {
    "changed": false,
    "msg": "EOF on stream; last 100 lines received:\nbash: /usr/bin/python: No such file or directory",
    "unreachable": true
}

Adding interpreter_python = /usr/bin/python3 into ansible.cfg doesn't help.

[s1113950]

@shinebayar-g ansible python interpreter discovery isn't in 0.2.9. Your error looks more like Mitogen is picking the wrong version of Python to use. I recommend either using the latest git hash of Mitogen master until #715 lands, or trying setting ansible_interpreter_python rather than interpreter_python to fix this issue.

[hackatoo]

@jasminehung Do you still have problems with connecting mitogen to Windows Host or you resolved it? If yes, please share how. Today i was trying to test it, but had exact same error, as you 2 years ago.

Unhandled error in Python interpreter discovery for host 209.58.171.119: EOF on stream; last 100 lines received: File "<string>", line 1 import codecs,os,sys;_=codecs.decode;exec(_(_(eNqVkl1PwjAUhu/3K7w7bVxGO0iMi0skgGIiH1nQxSBZ9tFJA7RLN 5j46+2YgYEXxrs+fd9zzrv1eKbvytzKeMYQNpRZNoinVxpSqVYIO0Z1TraZjYhJCcEn9swmKa3SmuO1zBnymqCa4Deh1FAP3Ipwo8fPycJ1oR+qkgsIRdKU7MXcsRd3Q G8PSr7PLfbJ4m0RRmumq1rbXLUiLlrZvlhKAc6F49JgWzfw7/H63iY2PSbYMZVzKeZOu6qsWjq6hokdV/oWpm+z4WQcPHdfxr3hoB88eJNR4Hvd6XTgwcIFCoe/cfR3v cdXooXz6LVH4xqdC+Y5XgPa8EJ+MOEsw3gVFlLeTxXf9VkeO7TTabcxYEP3KhUvGKImjJ5mE0LIuwCdIZaJ/lBs9NwAVTuQyIwJ/fKgIsCWYmGC6A0lHWzCF890pzRzT z7fhDKCai3S7GdA73Cun/rCXf7l/p2Snqc8LpGNvwE1QeLn.encode(),base64),zip))

I've added symlink from /usr/bin/python to python binary on my host and enviromental variables to PATH. But still getting this error. Have Python 3.10.12 on the client side with Mitogen 0.3.3 on the linux instance.

[jasminehung]

@hackatoo Sadly I didn't find any workable solution to this error

[electronick]

I decided to give mitogen a try and after installation I've got the same error when trying to execute any existing ansible playbook I had. I'm on MacOS and in my case the issue was connected to string interpolation in ansible_ssh_common_args. I had the following in my group_vars for gatewayed hosts

ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q {{ bastion_user }}@{{ bastion_public_hostname }}"' 

and I had to move to static user and host without interpolation in ansible_ssh_common_args for mitogen to work. After I removed interpolation from ansible_ssh_common_args, it worked!