pypsexec icon indicating copy to clipboard operation
pypsexec copied to clipboard

When connecting to Windows Server 2008 R2, connection never closed eventhough the job is done successfully

Open FaisalDefry opened this issue 5 years ago • 17 comments

I tried to create simple psexec script :

from pypsexec.client import Client

server = "windows2008.intra.com"
username = "Administrator"
password = "generic-password"
executable = "powershell.exe"
arguments = 'ls > C:\windows\temp\dir.txt'

def raw_string(s):
    if isinstance(s, str):
        s = s.encode('string-escape')
    elif isinstance(s, unicode):
        s = s.encode('unicode-escape')
    return s

c = Client(server, username=username, password=password,
           encrypt=False)

c.connect()
try:
    c.create_service()
    result = c.run_executable(executable, arguments=raw_string(arguments))
finally:
    c.remove_service()
    c.disconnect()

print("STDOUT:\n%s" % result[0].decode('utf-8') if result[0] else "")
print("STDERR:\n%s" % result[1].decode('utf-8') if result[1] else "")
print("RC: %d" % result[2])`

The code is always working on windows 2012 or later but when connecting to Windows Server 2008 R2, connection never closed eventhough the job is done successfully ( file dir.txt created).

It will always stuck in this line if i run the program with verbose :

...
# /usr/lib/python2.7/site-packages/pypsexec/paexec.pyc matches /usr/lib/python2.7/site-packages/pypsexec/paexec.py
import pypsexec.paexec # precompiled from /usr/lib/python2.7/site-packages/pypsexec/paexec.pyc
# /usr/lib/python2.7/site-packages/pypsexec/pipe.pyc matches /usr/lib/python2.7/site-packages/pypsexec/pipe.py
import pypsexec.pipe # precompiled from /usr/lib/python2.7/site-packages/pypsexec/pipe.pyc
# /usr/lib/python2.7/site-packages/pypsexec/scmr.pyc matches /usr/lib/python2.7/site-packages/pypsexec/scmr.py
import pypsexec.scmr # precompiled from /usr/lib/python2.7/site-packages/pypsexec/scmr.pyc
# /usr/lib/python2.7/site-packages/pypsexec/rpc.pyc matches /usr/lib/python2.7/site-packages/pypsexec/rpc.py
import pypsexec.rpc # precompiled from /usr/lib/python2.7/site-packages/pypsexec/rpc.pyc
# /usr/lib64/python2.7/Queue.pyc matches /usr/lib64/python2.7/Queue.py
import Queue # precompiled from /usr/lib64/python2.7/Queue.pyc
# /usr/lib64/python2.7/encodings/utf_16_le.pyc matches /usr/lib64/python2.7/encodings/utf_16_le.py
import encodings.utf_16_le # precompiled from /usr/lib64/python2.7/encodings/utf_16_le.pyc
# /usr/lib64/python2.7/encodings/ascii.pyc matches /usr/lib64/python2.7/encodings/ascii.py
import encodings.ascii # precompiled from /usr/lib64/python2.7/encodings/ascii.pyc
# /usr/lib64/python2.7/encodings/latin_1.pyc matches /usr/lib64/python2.7/encodings/latin_1.py
import encodings.latin_1 # precompiled from /usr/lib64/python2.7/encodings/latin_1.pyc
# /usr/lib64/python2.7/encodings/string_escape.pyc matches /usr/lib64/python2.7/encodings/string_escape.py
import encodings.string_escape # precompiled from /usr/lib64/python2.7/encodings/string_escape.pyc 

So currently the only workaround that i can think of is putting "timeout_seconds" so the program will close when the task is done, but i need to guess how much time the program is need to complete the task specified in argument before force closing.

Did i do something wrong ?

Thank you in advance

FaisalDefry avatar Aug 19 '19 09:08 FaisalDefry

I'll have to play around with this to try and replicate it, are you saying that the remote process has finished? The file may be present but checking if the PAExec process is still installed/running or if the actual powershell.exe process is still running would be great information to know.

I'm curious about your raw_string function, what is the reason for calling the .encode() function on your string objects?

jborean93 avatar Aug 20 '19 20:08 jborean93

Hi, sure i'll provide you more info tomorrow.

About the raw_string function, if i not use that somehow string 'C:\windows\temp\dir.txt' is interpreted as 'C:\windows emp\dir.txt' by the c.run_executable function eventhough i already defined it on single quote. Looks like the \t is interpreted as a "tab". I can escape it by define the string as 'C:\windows\temp\dir.txt', but anyone who not aware about that will get the same problem.

Actually i'm pretty new on python so i don't know if there's any simpler way to do that.

FaisalDefry avatar Aug 22 '19 13:08 FaisalDefry

Ah ok, that's because \ is an escape sequence in Python. There are 2 ways around this, you can either;

  • Double up and escape the escape sequence like "C:\\temp\\file.txt", or
  • Prefix the string with r like r"C:\temp\file.txt".

The latter is fine for simple stuff but if you then start needing escaping sequences like \n then the first option is more flexible. So in your case you want to do

arguments = 'ls > C:\\windows\\temp\\dir.txt'

Let me know when you get that info I was asking for and we can try and identify what is actually happening.

jborean93 avatar Aug 23 '19 06:08 jborean93

Hi, sorry for the late response. it seems PAExec process is still running along with the Powershell.exe process.

By the way, not sure if this info will be any help. Earlier i encounter the "remove_service throw exception" issue like in https://github.com/jborean93/pypsexec/issues/16 on both Windows 2012 and Windows 2008, when i add your workaround, the script work fine again on Windows 2012, but still hanging like before on Windows 2008.

FaisalDefry avatar Aug 26 '19 06:08 FaisalDefry

it seems PAExec process is still running along with the Powershell.exe process.

Ok something is keeping the process alive which is why pypsexec is escaping, can you try doing the following?

import base64
from pypsexec.client import Client

server = "windows2008.intra.com"
username = "Administrator"
password = "generic-password"
executable = "powershell.exe"

# -EncodedCommand expects a UTF-16 encoded string
b_ps_command = u"ls > C:\\windows\\temp\\dir.txt".encode("utf-16-le")
arguments = "-EncodedCommand %s" % base64.b64encode(b_ps_command).decode("utf-8")

c = Client(server, username=username, password=password,
           encrypt=False)

c.connect()
try:
    c.create_service()
    result = c.run_executable(executable, arguments=arguments)
finally:
    c.remove_service()
    c.disconnect()

print("STDOUT:\n%s" % result[0].decode('utf-8') if result[0] else "")
print("STDERR:\n%s" % result[1].decode('utf-8') if result[1] else "")
print("RC: %d" % result[2])`

This way you are running an actual PowerShell command that has a definitive endpoint whereas just running powershell.exe could be trying to run an interactive process and waiting for more input hence the hang.

By the way, not sure if this info will be any help. Earlier i encounter the "remove_service throw exception" issue like in #16 on both Windows 2012 and Windows 2008, when i add your workaround, the script work fine again on Windows 2012, but still hanging like before on Windows 2008.

Makes some sense, if the process is still running as well as the PAExec service then I would expect this error. The service is marked for deletion and when it stops it would be removed but until it is it would just be marked for deletion.

jborean93 avatar Aug 26 '19 10:08 jborean93

can you try doing the following?

Hi, i got this following errors both on python2 or python3

(TMP) [root@ansible-vcenter tmp]# python2 pstest.py 
Traceback (most recent call last):
  File "pstest.py", line 20, in <module>
    result = c.run_executable(executable, arguments=arguments)
  File "/usr/lib/python2.7/site-packages/pypsexec/client.py", line 291, in run_executable
    settings['arguments'] = self._encode_string(arguments)
  File "/usr/lib/python2.7/site-packages/pypsexec/client.py", line 424, in _encode_string
    return string.encode('utf-16-le') if string else None
AttributeError: 'list' object has no attribute 'encode'


(TMP) [root@ansible-vcenter tmp]# python pstest.py 
Traceback (most recent call last):
  File "pstest.py", line 20, in <module>
    result = c.run_executable(executable, arguments=arguments)
  File "/tmp/TMP/lib/python3.6/site-packages/pypsexec/client.py", line 291, in run_executable
    settings['arguments'] = self._encode_string(arguments)
  File "/tmp/TMP/lib/python3.6/site-packages/pypsexec/client.py", line 424, in _encode_string
    return string.encode('utf-16-le') if string else None
AttributeError: 'list' object has no attribute 'encode'

FaisalDefry avatar Aug 29 '19 06:08 FaisalDefry

Sorry I assumed arguments took a list but it’s actually just a string, change the arguments line to

arguments = "-EncodedCommand %s" % base64.b64encode(b_ps_command).decode("utf-8")

jborean93 avatar Aug 29 '19 06:08 jborean93

Hi it's still stuck on

---

import 'pypsexec.rpc' # <_frozen_importlib_external.SourceFileLoader object at 0x7f41fbebcef0>
import 'pypsexec.scmr' # <_frozen_importlib_external.SourceFileLoader object at 0x7f41fbebc3c8>
# trying /tmp/queue.cpython-36m-x86_64-linux-gnu.so
# trying /tmp/queue.abi3.so
# trying /tmp/queue.so
# trying /tmp/queue.py
# trying /tmp/queue.pyc
# trying /tmp/TMP/lib64/python3.6/queue.cpython-36m-x86_64-linux-gnu.so
# trying /tmp/TMP/lib64/python3.6/queue.abi3.so
# trying /tmp/TMP/lib64/python3.6/queue.so
# trying /tmp/TMP/lib64/python3.6/queue.py
# trying /tmp/TMP/lib64/python3.6/queue.pyc
# trying /tmp/TMP/lib64/python3.6/lib-dynload/queue.cpython-36m-x86_64-linux-gnu.so
# trying /tmp/TMP/lib64/python3.6/lib-dynload/queue.abi3.so
# trying /tmp/TMP/lib64/python3.6/lib-dynload/queue.so
# trying /tmp/TMP/lib64/python3.6/lib-dynload/queue.py
# trying /tmp/TMP/lib64/python3.6/lib-dynload/queue.pyc
# trying /usr/lib64/python3.6/queue.cpython-36m-x86_64-linux-gnu.so
# trying /usr/lib64/python3.6/queue.abi3.so
# trying /usr/lib64/python3.6/queue.so
# trying /usr/lib64/python3.6/queue.py
# /usr/lib64/python3.6/__pycache__/queue.cpython-36.pyc matches /usr/lib64/python3.6/queue.py
# code object from '/usr/lib64/python3.6/__pycache__/queue.cpython-36.pyc'
import 'queue' # <_frozen_importlib_external.SourceFileLoader object at 0x7f41fbebc668>
import 'pypsexec.client' # <_frozen_importlib_external.SourceFileLoader object at 0x7f4201b25978>
# trying /tmp/TMP/lib64/python3.6/encodings/utf_16_le.cpython-36m-x86_64-linux-gnu.so
# trying /tmp/TMP/lib64/python3.6/encodings/utf_16_le.abi3.so
# trying /tmp/TMP/lib64/python3.6/encodings/utf_16_le.so
# trying /tmp/TMP/lib64/python3.6/encodings/utf_16_le.py
# /tmp/TMP/lib64/python3.6/encodings/__pycache__/utf_16_le.cpython-36.pyc matches /tmp/TMP/lib64/python3.6/encodings/utf_16_le.py
# code object from '/tmp/TMP/lib64/python3.6/encodings/__pycache__/utf_16_le.cpython-36.pyc'
import 'encodings.utf_16_le' # <_frozen_importlib_external.SourceFileLoader object at 0x7f41fbe56e80>

By the way i got new finding, if i upgrade the powershell version using your ps1 script https://github.com/jborean93/ansible-windows/blob/master/scripts/Upgrade-PowerShell.ps1 . The pypsexec script works normally. However there's many case where upgrading powershell is not an option because it need rebooting the system. So if you have other workaround, that would be really appreciated.

FaisalDefry avatar Aug 29 '19 06:08 FaisalDefry

Hi, i just wondering is there any update on this? Thank you

FaisalDefry avatar Sep 18 '19 05:09 FaisalDefry

With the information provided I can't tell what could be the issue. If the process is still running on the remote side then we need to figure out why that still is. Does it only happen with PowerShell, is it only with certain commands. Can you run the same commands locally, what about through psexec itself.

jborean93 avatar Sep 18 '19 05:09 jborean93

It happens on all commands, even on simple command like “hostname”.

Btw, i found out something again. When using psexec or winexe, the hanging process would exit normally if we press “enter” key on the terminal.

So like it’s waiting for keypress or something.

FaisalDefry avatar Sep 18 '19 06:09 FaisalDefry

So did you try that example I gave you that used -EncodedCommand?

jborean93 avatar Sep 18 '19 07:09 jborean93

So did you try that example I gave you that used -EncodedCommand?

Yes, it's still stuck.

So the workaround that i've found is :

  • Upgrade powershell and .net version (https://github.com/jborean93/ansible-windows/blob/master/scripts/Upgrade-PowerShell.ps1). But it may not ideal on some situation when rebooting system is impossible.

  • Use "expect" scripting to simulate "enter" keypress.

FaisalDefry avatar Sep 18 '19 07:09 FaisalDefry

Hmm ok, it's weird that I haven't encountered this problem before but I will try and find some time to look into it again. You can set stdin=b"\r\n" as a kwarg to send a newline over the input pipe, that might simulate the same thing as running through expect.

jborean93 avatar Sep 18 '19 07:09 jborean93

I came accross this issue on a couple of servers with psexec and powershell. The problem is that powershell is waiting after the execution for some input like a single Newline to proceed. You can fix this by starting powershell with the parameter -InputFormat none So powershell will no longer wait for input.

walter-ebner avatar Dec 11 '19 15:12 walter-ebner

@walter-ebner Only the best people come back to a thread with the solution after solving it, mad props to you magnificent gentleman! I was having this exact issue on my Windows 7 and Windows Server 2008 targets and -InputFormat None did the trick!! Thank you!

@jborean93 I think this deserves a place in the docs right beside encrypt=False being necessary for Win7/2008, as this issue got me well and truly hard-stuck until I finally found this thread, and it relates to those same specific OSes. 😅 In any case, thank you for this great piece of software!

johanwintgens-ls avatar Mar 07 '23 15:03 johanwintgens-ls

Happened to me too. I wrote a Python script that works with several hundred computers (W10 mostly) running remote tasks with pypsexec, and on some of them (just a few of hundreds) running PowerShell from pypsexec caused the script to hang (until it reached the timeout [timeout_seconds]) after running the command.

Even simple commands like PowerShell -Command (gwmi Win32_Processor).Name would hang on some computers (for example on two Windows 7 ones) until timeout. I'm now using the equivalent of PowerShell -InputFormat None -Command ... instead and it works flawlessly.

NeoPolus avatar Apr 20 '23 06:04 NeoPolus