jborean93
Timeout on RunspacePool
Using 1.0.0b1
After (roughly) two minutes of inactivity, the RunspacePool will no longer allow sending commands. Creating a new pipeline will still work, but pipeline.invoke() will crash. Furthermore, it hangs the Python program and remains connected on the Windows host for a while.
Here is my example script to reproduce: test.py
import time
from psrp import WSManInfo, SyncRunspacePool
import psrp
hostname = "example.com"
username = "user"
password = "pass"
conn = WSManInfo(
hostname,
username=username,
password=password,
)
with SyncRunspacePool(conn) as rp:
print("Starting runspace pool...")
print(rp._pool.runspace_pool_id)
ps = psrp.SyncPowerShell(rp)
output = ps.add_script("date").add_command("Out-String").invoke()[0].strip()
print(output)
sleep_time = 130
time.sleep(sleep_time)
print(f"Slept {sleep_time}")
ps = psrp.SyncPowerShell(rp)
output = ps.add_script("date").add_command("Out-String").invoke()
print(output[0])
Even after the script ends (hangs or is forcibly killed), the state of the runspace still shows as connected on Windows:
PS C:\Users\Administrator> Get-WSManInstance -ConnectionURI 'http://localhost:5985/wsman' -ResourceURI shell -Enumerate | Format-Table ShellId,ClientIp,State,ShellRunTime,ShellInactivity
ShellId ClientIP State ShellRunTime ShellInactivity
------- -------- ----- ------------ ---------------
BE79B3CD-A866-4593-B140-21A8E154DDE8 192.168.181.1 Connected P0DT0H9M27S P0DT0H0M6S
Finally, here is my traceback:
Traceback
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_exceptions.py", line 10, in map_exceptions
yield
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_backends\sync.py", line 126, in read
return self._sock.recv(max_bytes)
~~~~~~~~~~~~~~~^^^^^^^^^^^
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "F:\Documents\GitHub\Devious-WinRM\src\devious_winrm\test.py", line 26, in <module>
output = ps.add_script("date").add_command("Out-String").invoke()
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_sync.py", line 1137, in invoke
output_task = self.invoke_async(
input_data=input_data,
output_stream=output_stream,
buffer_input=buffer_input,
)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_sync.py", line 1634, in invoke_async
return super().invoke_async(input_data, output_stream, completed, buffer_input)
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_sync.py", line 1204, in invoke_async
connection.command(self._pipeline.pipeline_id)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_connection\wsman.py", line 524, in command
resp = self._connection.post(self._shell.data_to_send())
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_io\wsman.py", line 1064, in post
response = self._http.post(
self.connection_uri,
...<4 lines>...
extensions=ext,
)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 1132, in post
return self.request(
~~~~~~~~~~~~^
"POST",
^^^^^^^
...<11 lines>...
extensions=extensions,
^^^^^^^^^^^^^^^^^^^^^^
)
^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 814, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 901, in send
response = self._send_handling_auth(
request,
...<2 lines>...
history=[],
)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
response = self._send_handling_redirects(
request,
follow_redirects=follow_redirects,
history=history,
)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
response = self._send_single_request(request)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
response = transport.handle_request(request)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_io\wsman.py", line 679, in handle_request
resp = self._connection.handle_request(req)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\connection.py", line 103, in handle_request
return self._connection.handle_request(request)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\http11.py", line 133, in handle_request
raise exc
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\http11.py", line 111, in handle_request
) = self._receive_response_headers(**kwargs)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\http11.py", line 176, in _receive_response_headers
event = self._receive_event(timeout=timeout)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\http11.py", line 212, in _receive_event
data = self._network_stream.read(
self.READ_NUM_BYTES, timeout=timeout
)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_backends\sync.py", line 124, in read
with map_exceptions(exc_map):
~~~~~~~~~~~~~~^^^^^^^^^
File "C:\Users\Pablo\AppData\Local\Programs\Python\Python313\Lib\contextlib.py", line 162, in __exit__
self.gen.throw(value)
~~~~~~~~~~~~~~^^^^^^^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_exceptions.py", line 14, in map_exceptions
raise to_exc(exc) from exc
httpcore.ReadError: [WinError 10054] An existing connection was forcibly closed by the remote host
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "F:\Documents\GitHub\Devious-WinRM\src\devious_winrm\test.py", line 17, in <module>
with SyncRunspacePool(conn) as rp:
~~~~~~~~~~~~~~~~^^^^^^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_sync.py", line 519, in __exit__
self.close()
~~~~~~~~~~^^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_sync.py", line 689, in close
connection.close()
~~~~~~~~~~~~~~~~^^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_connection\wsman.py", line 497, in close
resp = self._connection.post(self._shell.data_to_send())
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_io\wsman.py", line 1064, in post
response = self._http.post(
self.connection_uri,
...<4 lines>...
extensions=ext,
)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 1132, in post
return self.request(
~~~~~~~~~~~~^
"POST",
^^^^^^^
...<11 lines>...
extensions=extensions,
^^^^^^^^^^^^^^^^^^^^^^
)
^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 814, in request
return self.send(request, auth=auth, follow_redirects=follow_redirects)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 901, in send
response = self._send_handling_auth(
request,
...<2 lines>...
history=[],
)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 929, in _send_handling_auth
response = self._send_handling_redirects(
request,
follow_redirects=follow_redirects,
history=history,
)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 966, in _send_handling_redirects
response = self._send_single_request(request)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpx\_client.py", line 1002, in _send_single_request
response = transport.handle_request(request)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\psrp\_io\wsman.py", line 679, in handle_request
resp = self._connection.handle_request(req)
File "F:\Documents\GitHub\Devious-WinRM\.venv\Lib\site-packages\httpcore\_sync\connection.py", line 101, in handle_request
raise ConnectionNotAvailable()
httpcore.ConnectionNotAvailable
Any updates @jborean93? If you have an idea as to what might be causing the issue let me know, I have some free time that I could use to squash this bug.
@1upbyte see https://github.com/jborean93/pypsrp/issues/21#issuecomment-614590892
This is not the behavior I'm experiencing on the beta release. In my case the RunspacePool doesn't become disconnected.
I did manage a workaround by having a thread in the background create a new Pipeline using the RunspacePool and sending an empty script every 60 seconds.
I'm not ignoring this sorry it is just a bit low in my priority list right now. The whole connection handling probably needs an overhaul and while I've got a POC branch that drops httpx in favour of using httpcore directly I've not had the time to properly have it deal with things like timeouts and the like.
