AWS SSM Sessions

[sempervictus]

Amazon Web Services provides conveniently privileged backdoors in the form of their SSM agents which do not require connectivity with the target instance, merely valid credentials to AWS' API. Due to this indirect "connection" paradigm, this mechanism can be used to control otherwise "air-gapped" targets.

At the user-facing level, this PR provides two abstractions to the session handler atop which it can operate a command session.

1. Script-execution abstraction and pickup via AWS' API. This approach abstracts asynchronous request/response parsing for SSM requests into an IO channel with which the AWS SSM client is then wrapped to emulate the expected Stream. The mechanism is rather raw and could use better error handling, retries on laggy output, and a threadsafe cursor implementation. There is a significant limitation with these sessions not present in normal stream-wise abstractions: a response limit of 2500 chars. This limitation can be overcome by utilizing an S3 bucket to store command output; however, due to the nature of access we seek to obtain, it would not only add to the logged event loads but retain the results of our TTPs in a "buffer" accessible to other people. This functionality can be added down the line in the form of S3 config options in the handler to be passed into the SSM client for command execution and acquisition of output.

2. WebSocket-based "direct" interaction using @zeroSteiner's marvelous Rex implementation for WebSocket channels providing the necessary hooks by which to effect relevant encoding and decoding of "higher-level" protocols. This modality is more "mature" (though somewhat less novel than the C2 abstraction) in terms of interface presented, and supports fully interactive, colorful sessions via the stream abstraction.

Testing: Gets sessions, provides command IO, leaves a bunch of log entries in CloudTrail (something to keep in mind for opsec considerations).

How to test:

1. Pull PR into your test framework and install the resulting bundle to pull in the SSM SDK
2. Start the updated console
3. use auxiliary/cloud/aws/enum_ssm
4. set SECRET_ACCESS_KEY to your secret key
5. set ACCESS_KEY_ID to your access key ID
6. set CreateSession false
7. exploit
8. verify how many and which systems you're about to shell (this thing is an autopwn on roids in the wrong environment)
9. set CreateSession true if there are valid targets and they are safe to shell (over TLS, but still worth considering)
10. exploit

Advanced users can test the payloads (unix^windows) directly via the handler and setting the provided datastore options.

Notes:

• In my testing, running exploit just keeps on getting more shells. Not sure if its related to my mutant framework setup, but worth testing for the effect upstream (and figuring out why it happens, if it does).
• Suggested mitigation for this threat vector (OOB privileged access permitting command and raw IO) is to uninstall the bloody agent and make sure it stays that way (or just build and use images which have never had it on them in the first place, nix cloud-init after boostrapping as well, while at it)
• The two session abstraction stacks this work presents can (and IMO should) be used to build C2 sessions and tramsports to permit comms with "third-party software" (among other things) :wink:

[wvu]

This is cool. I'm also a fan of SSM's port forwarding.

[sempervictus]

This is cool. I'm also a fan of SSM's port forwarding.

Why do you hate me so? :stuck_out_tongue: I was hoping to stop this @ a proper WS-based session and not deal with another meterssh monstrosity :smile:

[sempervictus]

@wvu: i've exposed the various infrastructure hooks you'd need to start digging into the port-forwarding thing (or even some custom SSM document that raises Pythagoras' ghost to haunt target systems). Going to see if i can get this WS session style working and commented-up so i can keep the PR to a sane size.

@zeroSteiner, @jmartin-r7 & rest of R7 team (and any brave community devs like @smashery, @OJ ): y'all might want to look into the (not exactly pretty) command-exec abstraction i made to cobble-together async dispatch of SSM commands and their retrieval into something that smells like a synchronous IO to the session handler. That code is technically-speaking a "real C2 abstraction" which might be used to pilot asymmetric or unpleasantly asynchronous IO. Its not the proper decoupling of sessions/handlers/payloads/arch/platform/runtime/os which would be ideal to have happen... but it looks to me to be a viable path forward for expanding session types into actual C2+ space and handling 3rd-party sessions (webshells seem like a good starting point). One thing that comes to mind working with this PR is that it would be really nice to have meterpreter/mettle run their comms over the STDIO channels such things let us use (especially the byte-wise ones vs line-wise) since then we could "handle the upgrade" framework-side pretty easily by consuming the existing channel and inserting our TLV encoder (base64 or whatever) at that layer... thoughts?

[sempervictus]

WebSocket shells work as of 43d746c:

(2023-01-03)09:28 (S:2 J:0)msf  exploit(multi/handler) > sessions 2
[*] Starting interaction with 2...


Shell Banner:
$
-----
          

$ whoami
whoami
ssm-user
$ pwd
pwd
/var/snap/amazon-ssm-agent/6312
sudo su
root@pwned-hostname:/var/snap/amazon-ssm-agent/6312# whoami
whoami
root

We might wanna clean-up that echo piece, but the technically-difficult parts should be all set now.

[jmartin-tech]

Thanks for the neat new toy. Please add a Gemfile.lock change required by the gemspec update.

diff --git a/Gemfile.lock b/Gemfile.lock
index 8a5a4356ae..703f795ad2 100644
--- a/Gemfile.lock
+++ b/Gemfile.lock
@@ -8,6 +8,7 @@ PATH
       aws-sdk-ec2
       aws-sdk-iam
       aws-sdk-s3
+      aws-sdk-ssm
       bcrypt
       bcrypt_pbkdf
       bson
@@ -146,6 +147,9 @@ GEM
       aws-sdk-core (~> 3, >= 3.165.0)
       aws-sdk-kms (~> 1)
       aws-sigv4 (~> 1.4)
+    aws-sdk-ssm (1.146.0)
+      aws-sdk-core (~> 3, >= 3.165.0)
+      aws-sigv4 (~> 1.1)
     aws-sigv4 (1.5.2)
       aws-eventstream (~> 1, >= 1.0.2)
     bcrypt (3.1.18)

Further review is still pending.

[sempervictus]

@jmartin-r7 - there's gonna be a few of those weird things since the repo into which i dump the working files is not the same as my framework, also why i cant actually run tidy apparently :confused:

For straightforward things like this or the tidy pass, could you please PR a commit with all the small changes into the source branch for this PR? I'll just merge the fixes in one go atop this and then we can deal with the thought-provoking stuff like "do we want keepalive there by default internally triggered by SsmChannel init or initiated externally by its consumer/caller" afterwards.

[sempervictus]

@zeroSteiner - how do you want to handle output/reporting/etc? Should i throw in a scanner mixin or do we just want a pretty Rex table with the contents of that JSON message? Anything past the cosmetics outstanding? Also, could you please msftidy the PR since my env isn't happy running the linter?

[sempervictus]

@smcintyre-r7: cant comment on the region piece, but that is a requirement in the API call. Updating the DS option to be required seems the most straightforward approach. Any issue w/ my doing that?

[sempervictus]

Regarding the exit bit - will need to dig into that, i normally ctrl+c my sessions to get out of them but its either my having missed some "end of session" message type in the WS proto or that the WS is independent of the shell invocation behind it... if its the latter, all ears on suggestions for how to handle a live session Stream with no backing shell to execute and return.

[sempervictus]

@smcintyre-r7 - re the exit bit: looks like close isn't working correctly, SsmChannel records the socket close, but then it seems that the underlying Channel tries to call opcode on a Hash:

[01/21/2023 00:15:59] [e(0)] core: SsmChannel got closed message b77c25d2-90ac-44c6-8637-10d9031ccecb
[01/21/2023 00:15:59] [e(0)] core: Thread Exception: WebSocketChannel(MY-IP->SSM-API-IP)  critical=false    source:
    /opt/metasploit4/msf4/lib/metasploit/framework/thread_factory_provider.rb:25:in `spawn'
    /opt/metasploit4/msf4/lib/rex/thread_factory.rb:22:in `spawn'
    /opt/metasploit4/msf4/lib/rex/proto/http/web_socket.rb:71:in `initialize'
    /opt/metasploit4/msf4/lib/rex/proto/http/web_socket/amazon_ssm.rb:153:in `initialize'
    /opt/metasploit4/msf4/lib/rex/proto/http/web_socket/amazon_ssm.rb:195:in `new'
    /opt/metasploit4/msf4/lib/rex/proto/http/web_socket/amazon_ssm.rb:195:in `to_ssm_channel'
    /opt/metasploit4/msf4/modules/auxiliary/cloud/aws/enum_ssm.rb:136:in `get_ssm_socket'
    /opt/metasploit4/msf4/modules/auxiliary/cloud/aws/enum_ssm.rb:103:in `block in run'
    /opt/metasploit4/msf4/modules/auxiliary/cloud/aws/enum_ssm.rb:87:in `each'
    /opt/metasploit4/msf4/modules/auxiliary/cloud/aws/enum_ssm.rb:87:in `run'
    /opt/metasploit4/msf4/lib/msf/base/simple/auxiliary.rb:180:in `job_run_proc'
    /opt/metasploit4/msf4/lib/msf/base/simple/auxiliary.rb:85:in `run_simple'
    /opt/metasploit4/msf4/lib/msf/base/simple/auxiliary.rb:96:in `run_simple'
    /opt/metasploit4/msf4/lib/msf/ui/console/command_dispatcher/auxiliary.rb:69:in `cmd_run'
    /opt/metasploit4/msf4/lib/rex/ui/text/dispatcher_shell.rb:581:in `run_command'
    /opt/metasploit4/msf4/lib/rex/ui/text/dispatcher_shell.rb:530:in `block in run_single'
    /opt/metasploit4/msf4/lib/rex/ui/text/dispatcher_shell.rb:524:in `each'
    /opt/metasploit4/msf4/lib/rex/ui/text/dispatcher_shell.rb:524:in `run_single'
    /opt/metasploit4/msf4/lib/rex/ui/text/resource.rb:69:in `load_resource'
    /opt/metasploit4/msf4/lib/msf/ui/console/command_dispatcher/resource.rb:73:in `block in cmd_resource'
    /opt/metasploit4/msf4/lib/msf/ui/console/command_dispatcher/resource.rb:52:in `each'
    /opt/metasploit4/msf4/lib/msf/ui/console/command_dispatcher/resource.rb:52:in `cmd_resource'
    /opt/metasploit4/msf4/lib/rex/ui/text/dispatcher_shell.rb:581:in `run_command'
    /opt/metasploit4/msf4/lib/rex/ui/text/dispatcher_shell.rb:530:in `block in run_single'
    /opt/metasploit4/msf4/lib/rex/ui/text/dispatcher_shell.rb:524:in `each'
    /opt/metasploit4/msf4/lib/rex/ui/text/dispatcher_shell.rb:524:in `run_single'
    /opt/metasploit4/msf4/lib/rex/ui/text/shell.rb:162:in `run'
    /opt/metasploit4/msf4/lib/metasploit/framework/command/console.rb:50:in `start'
    /opt/metasploit4/msf4/lib/metasploit/framework/command/base.rb:82:in `start'
    /opt/metasploit4/msf4/msfconsole:23:in `<main>' - NoMethodError undefined method `opcode' for {:header=>{:fin=>1, :opcode=>2, :masked=>0, :payload_len_sm=>126, :payload_len_md=>371}, :payload_data=>"\x00\x00\x00tchannel_closed\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x01\x85\xD2\xC1CP\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x867\x10\xD9\x03\x1C\xCE\xCB\xB7|%\xD2\x90\xACD\xC6\b\xA8\x84\xAA\xD9\x9F\xB5\xEC-u\x17'0\x1Dm\xF4\x0F\xBA>\xE8\xA8s\xBA\x855\xF7\x19\xAE\x06\x1E\xD6@\x00\x00\x00\x00\x00\x00\x00\xFB{\"MessageId\":\"b77c25d2-90ac-44c6-8637-10d9031ccecb\",\"CreatedDate\":\"2023-01-21T05:15:59.185Z\",\"DestinationId\":\"c3478803-4842-49e9-922b-d6f7d3e69c4f\",\"SessionId\":\"SESSION-ID\",\"MessageType\":\"channel_closed\",\"SchemaVersion\":1,\"Output\":\"\"}"}:Rex::Proto::Http::WebSocket::Frame
[01/21/2023 00:16:03] [d(0)] core: HistoryManager.pop_context name: :shell
[01/21/2023 00:16:04] [d(0)] core: monitor_rsock: EOF in rsock

That threaded stack trace is a mess, so my guess is that its coming from wsloop in case frame.header.opcode ... Am i running my close incorrectly or is there a bug in the WebSocket::Interface::Channel code?

[smcintyre-r7]

All issues I've noticed thus far appear to have been resolved with the exception of the exit command not working. I'll look into that one more thoroughly to see if it's a websocket issue or not.

[sempervictus]

Anecdotally, i think that exit thing is a bigger problem and we're leaking a resource somewhere. I left framework running all weekend on the current state, using the SSM sessions to test the cmd_shell transport prototype. That testing kills shells, which seem to leave something behind eventually slowing framework down visibly after ~36h running idle after the abuse (this may not be the case upstream, mine's a zombie patchwork quilt of funsies). I figured it would be another 40y before i had to say this, but, i think that "one of us is leaking something" :sweat_drops: :wink: Having said that, Murphy dictates that we'll find its actually down in Rex...

[smcintyre-r7]

Had some time to test this a bit more, opened sepervictus/metasploit-framework#34 to fix the exit session error, though I am still concerned there's a leak somewhere that will impact performance.

Next up is I'm noticing that this is exhibiting quite a few failures with the post/test/cmd_exec and post/test/file modules. That makes me concerned this may not work with quite a few post modules.

msf6 post(test/cmd_exec) > run

[!] SESSION may not be compatible with this module:
[!]  * incompatible session type: shell
[!]  * incompatible session platform: 
[*] Running against session -1
[*] Session type is shell and platform is 
[-] FAILED: should return the result of echo with single quotes
[-] FAILED: should return the result of echo with double quotes
[-] FAILED: should return the stderr output
[+] should return the result of echo
[-] FAILED: should return the full response after sleeping
[-] FAILED: should return the full response after sleeping
[+] should return the result of echo 10 times
[-] Passed: 2; Failed: 5
[*] Post module execution completed
msf6 post(test/cmd_exec) > use post/test/file 
msf6 post(test/file) > run


[!] SESSION may not be compatible with this module:
[!]  * incompatible session platform: 
[*] Running against session -1
[*] Session type is shell and platform is 
[+] should expand home
[+] should not expand non-isolated tilde
[-] FAILED: should not expand mid-string tilde
[-] FAILED: should not expand env vars with invalid naming
[-] FAILED: should expand multiple variables
[-] Passed: 2; Failed: 3
[*] Post module execution completed
msf6 post(test/file) > 

[sempervictus]

Given that the shell colorizes and seems to be a full PTY, its definitely not your standard command-shell... might also be why my in-shell-TLV-transport thing is barfing.

[smcintyre-r7]

Alright, here's an updated list of outstanding tasks that need to be resolved:

• [x] The datastore options with the payload modules need to be consistent with enum_ssm
• [ ] The #platform attribute needs to be set on all of the sessions
  • I noticed that both the payload handler and the enum_ssm modules do not set this appropriately. That means the framework can't tell what post modules should be compatible with it.
  • Pretty sure this is due to using Msf::Sessions::CommandShell as the session type and not Msf::Sessions::CommandShellUnix / Msf::Sessions::CommandShellWindows
• [ ] The handler needs to print [*] Started bind AWS SSM handler against exactly once when invoked from to_handler in the payload modules
• [ ] The handler needs to not continuously open sessions on the same host. After once it should probably stop, but more importantly the behavior should be consistent with the other bind handlers.
• [x] The UUID bit needs to be sorted, if we're not sure, I'd lean towards just using the existing definition.
• [ ] The post/test/file module is reporting that most tests are failing, these tests need to pass so the session is compatible with post modules
• [ ] The post/test/cmd_exec module is also reporting that most tests are failing, these test also need to pass so the session is compatible with post modules
• [x] The unit tests need to pass, I think this is mostly linting (fixed in 687e82a9ed2fd6b47f2b7aa9194b211cfc93fc56)

[sempervictus]

Alright, here's an updated list of outstanding tasks that need to be resolved:

* [ ]  The datastore options with the payload modules need to be consistent with enum_ssm

* [ ]  The `#platform` attribute needs to be set on all of the sessions
  
  * I noticed that both the payload handler and the `enum_ssm` modules do not set this appropriately. That means the framework can't tell what post modules should be compatible with it.
  * Pretty sure this is due to using `Msf::Sessions::CommandShell` as the session type and not `Msf::Sessions::CommandShellUnix` / `Msf::Sessions::CommandShellWindows`

* [ ]  The handler needs to not print `[*] Started bind AWS SSM handler against` exactly once when invoked from `to_handler` in the payload modules

* [ ]  The handler needs to not continuously open sessions on the same host. After once it should probably stop, but more importantly the behavior should be consistent with the other bind handlers.

* [ ]  The [UUID bit](https://github.com/rapid7/metasploit-framework/pull/17430#discussion_r1093740733) needs to be sorted, if we're not sure, I'd lean towards just using the existing definition.

* [ ]  The `post/test/file` module is reporting that most tests are failing, these tests need to pass so the session is compatible with post modules

* [ ]  The `post/test/cmd_exec` module is also reporting that most tests are failing, these test also need to pass so the session is compatible with post modules

* [ ]  The unit tests need to pass, I think this is mostly linting

Thank you for itemizing that. Could you please run the linters and PR their demands? I can't get it to execute - some crazy stack traces i've not fully parsed-out. I can tackle the handler, sessions, and datastore bits this weekend, could you take a look at the session IO piece - my reasonably well informed guess is that the file and command failures are due to the extra non-printable characters coming in over the session (notice how it color-code ubuntu SSM shells in the session context?), or some sort of TTY-like behavior. If not for the massive data amplification of sending WS frames per-char, i'd consider looking at this as a character-wise (vs line-wise) interface.

[sempervictus]

• DS options should be good

• Platforms should now be correct

• Handler is still running as a job for some reason, repeatedly creating sessions :confused:

[sempervictus]

is this still running the handler repeatedly in upstream framework? anyone have any thoughts on what the deuce could be causing that if so?

[adfoster-r7]

Looks like the download command is broken on the windows ssm powershell session

Setup

Create a file and the download command should work:

msf6 payload(cmd/windows/powershell/x64/powershell_reverse_tcp) > sessions -i -1

# create the file
PS C:\Users\vagrant> cd c:/
PS C:\> mkdir temp
PS C:\> cd temp
PS C:\temp> echo 'abc' > foo

Existing powershell_reverse_tcp working 🟢

Expected behavior from a powershell session:

# Download and verify b64 response:
PS C:\temp> download c:/temp/foo foo
[*] Download c:/temp/foo => foo

From: /Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:777 Msf::Post::File#_read_file_powershell_fragment:

    763: def _read_file_powershell_fragment(filename, chunk_size, offset = 0)
    764:   pwsh_code = <<~PSH
    765:     $mstream = New-Object System.IO.MemoryStream;
    766:     $gzipstream = New-Object System.IO.Compression.GZipStream($mstream, [System.IO.Compression.CompressionMode]::Compress);
    767:     $get_bytes = [System.IO.File]::ReadAllBytes(\"#{filename}\")[#{offset}..#{offset + chunk_size - 1}];
    768:     $gzipstream.Write($get_bytes, 0, $get_bytes.Length);
    769:     $gzipstream.Close();
    770:     [System.Convert]::ToBase64String($mstream.ToArray());
    771:   PSH
    772:   b64_data = cmd_exec(pwsh_code)
    773:   return nil if b64_data.empty?
    774: 
    775:   require 'pry-byebug'; binding.pry
    776: 
 => 777:   uncompressed_fragment = Zlib::GzipReader.new(StringIO.new(Base64.decode64(b64_data))).read
    778:   return uncompressed_fragment
    779: end

[1] pry(#<Msf::Sessions::CommandShell::FileTransfer>)> b64_data
=> "H4sIAAAAAAAEAPv/L5EhiSGZgZeBiwEAS353vAwAAAA="
[2] pry(#<Msf::Sessions::CommandShell::FileTransfer>)> continue
[+] Done

Local disk as expected:

xxd foo 
00000000: fffe 6100 6200 6300 0d00 0a00            ..a.b.c.....

New SSM Windows session failing 🔴

In comparison to the ssm powershell shell, which echos out the input command first - breaking the parsing:

PS C:\temp>download c:/temp/foo foo

[*] Download c:/temp/foo => foo

From: /Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:777 Msf::Post::File#_read_file_powershell_fragment:

    763: def _read_file_powershell_fragment(filename, chunk_size, offset = 0)
    764:   pwsh_code = <<~PSH
    765:     $mstream = New-Object System.IO.MemoryStream;
    766:     $gzipstream = New-Object System.IO.Compression.GZipStream($mstream, [System.IO.Compression.CompressionMode]::Compress);
    767:     $get_bytes = [System.IO.File]::ReadAllBytes(\"#{filename}\")[#{offset}..#{offset + chunk_size - 1}];
    768:     $gzipstream.Write($get_bytes, 0, $get_bytes.Length);
    769:     $gzipstream.Close();
    770:     [System.Convert]::ToBase64String($mstream.ToArray());
    771:   PSH
    772:   b64_data = cmd_exec(pwsh_code)
    773:   return nil if b64_data.empty?
    774: 
    775:   require 'pry-byebug'; binding.pry
    776: 
 => 777:   uncompressed_fragment = Zlib::GzipReader.new(StringIO.new(Base64.decode64(b64_data))).read
    778:   return uncompressed_fragment
    779: end

[1] pry(#<Msf::Sessions::CommandShell::FileTransfer>)> b64_data
=> "PS C:\\temp>\r\e[1A\e[?25h\e[?25l\r\n $mstream = New-Object System.IO.MemoryStream;\r\n $gzipstream = New-Object System.IO.Compression.GZipStream($mstream, [System.IO.Compression.CompressionMode]::Compress);\r\n\e[?25hPS C:\\temp> $get_bytes = [System.IO.File]::ReadAllBytes(\"c:/temp/foo\")[0..65535];\e[?25l\r\n\e[?25hPS C:\\temp> $gzipstream.Write($get_bytes, 0, $get_bytes.Length);\e[?25l\r\n $gzipstream.Close();\r\n [System.Convert]::ToBase64String($mstream.ToArray());\r\nH4sIAAAAAAAEAPv/L5GBl4GLAQCjdAW8CAAAAA==\e[?25h\e[?25l\r\n\r\n '"
[2] pry(#<Msf::Sessions::CommandShell::FileTransfer>)> continue
[-] Session manipulation failed: not in gzip format ["/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:777:in `initialize'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:777:in `new'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:777:in `_read_file_powershell_fragment'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:753:in `block in _read_file_powershell'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:752:in `loop'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:752:in `_read_file_powershell'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:481:in `read_file'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:430:in `cmd_download'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:617:in `run_builtin_cmd'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:605:in `run_single'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:769:in `_interact_stream'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:745:in `block in _interact'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/shell/history_manager.rb:49:in `with_context'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:744:in `_interact'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/interactive.rb:53:in `interact'", "/Users/user/Documents/code/metasploit-framework/lib/msf/ui/console/command_dispatcher/core.rb:1682:in `cmd_sessions'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:581:in `run_command'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:530:in `block in run_single'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:524:in `each'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:524:in `run_single'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/shell.rb:162:in `run'", "/Users/user/Documents/code/metasploit-framework/lib/metasploit/framework/command/console.rb:48:in `start'", "/Users/user/Documents/code/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'", "./msfconsole:23:in `<main>'"]

[github-actions[bot]]

Thanks for your pull request! Before this can be merged, we need the following documentation for your module:

• Writing Module Documentation
• Template
• Examples

[adfoster-r7]

Not a blocker, as it might be an existing issue with the prompt/shell history tracking

It looks like swapping between the newly opened session and the top level command prompt breaks the pry history:

# 1) View existing history
msf6 auxiliary(cloud/aws/enum_ssm) > history
set .... ...
set .... ...
run

# 2) Interact with the session and background it
msf6 auxiliary(cloud/aws/enum_ssm) > sessions -i -1
[*] Starting interaction with 1...

whoami
ssm-user

# 3) View the history, to see it's broken:
background

Background session 1? [y/N]  y
msf6 auxiliary(cloud/aws/enum_ssm) > history
1  history

Just marking this down here for now, will have test against a normal command session to see if that's a framework issue or an issue with the session

[sempervictus]

Looks like the download command is broken on the windows ssm powershell session

Setup

Create a file and the download command should work:

msf6 payload(cmd/windows/powershell/x64/powershell_reverse_tcp) > sessions -i -1

# create the file
PS C:\Users\vagrant> cd c:/
PS C:\> mkdir temp
PS C:\> cd temp
PS C:\temp> echo 'abc' > foo

Existing powershell_reverse_tcp working green_circle

Expected behavior from a powershell session:

# Download and verify b64 response:
PS C:\temp> download c:/temp/foo foo
[*] Download c:/temp/foo => foo

From: /Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:777 Msf::Post::File#_read_file_powershell_fragment:

    763: def _read_file_powershell_fragment(filename, chunk_size, offset = 0)
    764:   pwsh_code = <<~PSH
    765:     $mstream = New-Object System.IO.MemoryStream;
    766:     $gzipstream = New-Object System.IO.Compression.GZipStream($mstream, [System.IO.Compression.CompressionMode]::Compress);
    767:     $get_bytes = [System.IO.File]::ReadAllBytes(\"#{filename}\")[#{offset}..#{offset + chunk_size - 1}];
    768:     $gzipstream.Write($get_bytes, 0, $get_bytes.Length);
    769:     $gzipstream.Close();
    770:     [System.Convert]::ToBase64String($mstream.ToArray());
    771:   PSH
    772:   b64_data = cmd_exec(pwsh_code)
    773:   return nil if b64_data.empty?
    774: 
    775:   require 'pry-byebug'; binding.pry
    776: 
 => 777:   uncompressed_fragment = Zlib::GzipReader.new(StringIO.new(Base64.decode64(b64_data))).read
    778:   return uncompressed_fragment
    779: end

[1] pry(#<Msf::Sessions::CommandShell::FileTransfer>)> b64_data
=> "H4sIAAAAAAAEAPv/L5EhiSGZgZeBiwEAS353vAwAAAA="
[2] pry(#<Msf::Sessions::CommandShell::FileTransfer>)> continue
[+] Done

Local disk as expected:

xxd foo 
00000000: fffe 6100 6200 6300 0d00 0a00            ..a.b.c.....

New SSM Windows session failing red_circle

In comparison to the ssm powershell shell, which echos out the input command first - breaking the parsing:

PS C:\temp>download c:/temp/foo foo

[*] Download c:/temp/foo => foo

From: /Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:777 Msf::Post::File#_read_file_powershell_fragment:

    763: def _read_file_powershell_fragment(filename, chunk_size, offset = 0)
    764:   pwsh_code = <<~PSH
    765:     $mstream = New-Object System.IO.MemoryStream;
    766:     $gzipstream = New-Object System.IO.Compression.GZipStream($mstream, [System.IO.Compression.CompressionMode]::Compress);
    767:     $get_bytes = [System.IO.File]::ReadAllBytes(\"#{filename}\")[#{offset}..#{offset + chunk_size - 1}];
    768:     $gzipstream.Write($get_bytes, 0, $get_bytes.Length);
    769:     $gzipstream.Close();
    770:     [System.Convert]::ToBase64String($mstream.ToArray());
    771:   PSH
    772:   b64_data = cmd_exec(pwsh_code)
    773:   return nil if b64_data.empty?
    774: 
    775:   require 'pry-byebug'; binding.pry
    776: 
 => 777:   uncompressed_fragment = Zlib::GzipReader.new(StringIO.new(Base64.decode64(b64_data))).read
    778:   return uncompressed_fragment
    779: end

[1] pry(#<Msf::Sessions::CommandShell::FileTransfer>)> b64_data
=> "PS C:\\temp>\r\e[1A\e[?25h\e[?25l\r\n $mstream = New-Object System.IO.MemoryStream;\r\n $gzipstream = New-Object System.IO.Compression.GZipStream($mstream, [System.IO.Compression.CompressionMode]::Compress);\r\n\e[?25hPS C:\\temp> $get_bytes = [System.IO.File]::ReadAllBytes(\"c:/temp/foo\")[0..65535];\e[?25l\r\n\e[?25hPS C:\\temp> $gzipstream.Write($get_bytes, 0, $get_bytes.Length);\e[?25l\r\n $gzipstream.Close();\r\n [System.Convert]::ToBase64String($mstream.ToArray());\r\nH4sIAAAAAAAEAPv/L5GBl4GLAQCjdAW8CAAAAA==\e[?25h\e[?25l\r\n\r\n '"
[2] pry(#<Msf::Sessions::CommandShell::FileTransfer>)> continue
[-] Session manipulation failed: not in gzip format ["/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:777:in `initialize'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:777:in `new'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:777:in `_read_file_powershell_fragment'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:753:in `block in _read_file_powershell'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:752:in `loop'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:752:in `_read_file_powershell'", "/Users/user/Documents/code/metasploit-framework/lib/msf/core/post/file.rb:481:in `read_file'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:430:in `cmd_download'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:617:in `run_builtin_cmd'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:605:in `run_single'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:769:in `_interact_stream'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:745:in `block in _interact'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/shell/history_manager.rb:49:in `with_context'", "/Users/user/Documents/code/metasploit-framework/lib/msf/base/sessions/command_shell.rb:744:in `_interact'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/interactive.rb:53:in `interact'", "/Users/user/Documents/code/metasploit-framework/lib/msf/ui/console/command_dispatcher/core.rb:1682:in `cmd_sessions'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:581:in `run_command'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:530:in `block in run_single'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:524:in `each'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/dispatcher_shell.rb:524:in `run_single'", "/Users/user/Documents/code/metasploit-framework/lib/rex/ui/text/shell.rb:162:in `run'", "/Users/user/Documents/code/metasploit-framework/lib/metasploit/framework/command/console.rb:48:in `start'", "/Users/user/Documents/code/metasploit-framework/lib/metasploit/framework/command/base.rb:82:in `start'", "./msfconsole:23:in `<main>'"]

I implemented a type of echo suppression which seems to work on the unix sessions - should we try to expand this to prevent the PSH echo as well?

[smcintyre-r7]

After spending additional days trying to fix the Windows SSM session, I have been unable to find a solution that can reliably address this issue.

The remote side using WinPTY poses a significant complication that is simply unaccounted for within our post API. I explored two solutions in an attempt to address this.

The echoing could be disabled. This is the approach taken with the Linux sessions through some stty and pipe trickery. Once the command has been run, the session no longer behaves as though it's within a TTY. I was unable to find a working solution to implement this but it's theoretically possible using the .NET API to invoke native methods. This in and of itself would likely pose a risk to the session by increasing the detection surface.

The PTY related characters could be filtered. This is the existing proposed solution however it simply doesn't work in all the necessary cases. I found that tests would fail intermittently because a chunk of data that was read from the socket didn't match the string that was to be filtered because it was incomplete. I attempted to account for this by using a line buffering approach. I took the data that was to be filtered and queued it by line. I then attempted to check each line as it was read after the entire line was read to see if it ended with the top queue item. This was further complicated by the fact that the remote end would not only echo the typed characters back but also reprint the prompt. Where I left off was in dealing with long lines that were being sent and received. On sending a large command to upload a file, the remote end would interrupt it by sending back the prompt which after stripping the PTY control characters simply couldn't be accounted for.

It's my recommendation that we completely remove support for the Windows platform until a significant amount of time has been invested in fixing the issues to make the session work as sessions are expected to work in Metasploit. At one point @adfoster-r7 had suggested we find a way to mark this as not working with any post modules which would be feasible if the meta-shell commands (such as download) can be blocked as well.

On Monday, I'll look into implementing the restrictions on the session and if I'm unsuccessful, I'll remove Windows support so we can advance the remaining work as is and get Linux support merged.

[sempervictus]

Thank you for deep diving the nonsense @smcintyre-r7. I owe you a new dry suit and air tank as i doubt any amount of sanitization will get that smell out. As i understand this, we're running a powershell environment which renders XML into something that looks like text atop a "pty" implementation which is aberrant in windows anyway through the websocket. We know the websocket works more or less correctly since we got the POSIX stuff to work, but the corner-cases of the above coupled with the odd re-printing of the prompt make full acceptance testing infeasible. I dont think its on us to fix MSFTs decisions - nobody here's paid nearly enough for that, but i also dont think we want to lose what could be a very useful form of interaction with a windows target environment due to instability. This was the problem with UDP shells as well: UDP's stateless, can just fail, and you're left sitting there beating your head against the wall (fine for hackers with no other way out, not great when you're sticking it into a product).

Could we "cheat" and instantiate cmd.exe instead of PSH on the other side using the AWS SSM command doc to avoid the rendering issue? IMO PSH belongs in blocks encoded on the commandline and staging binary payloads handing their own MM :wink:. Can we "gate" the windows option behind a dammit-i-know-what-im-doin-jim barrier to avoid selective redaction of session capabilities having to be implemented before this can land?

[smcintyre-r7]

FWIW I blame Amazon for the decision to use WinPTY which is the real culprit behind our woes not Microsoft.

I'll look at using cmd.exe again, but I don't suspect it'll change anything because it too will be running within WinPTY and have the same echoing, line break issues.

[sempervictus]

If this is what they're using, then it appears to not have been updated for 5y. I suppose we could "promote migration to a safer mechanism" by finding & disclosing -> publishing a vuln in that thing :innocent:. Disclosure timelines would slow things down, but eventually we'd be rid of the nonsense.

Also looks like the thing has a debug mode similar to how meterp is wired:

Debugging winpty
winpty comes with a tool for collecting timestamped debugging output. To use it:

Run winpty-debugserver.exe on the same computer as winpty.
Set the WINPTY_DEBUG environment variable to trace for the winpty.exe process and/or the process using libwinpty.dll.
winpty also recognizes a WINPTY_SHOW_CONSOLE environment variable. Set it to 1 to prevent winpty from hiding the console window.

I have a sneaking suspicion that the output cleanup you're dealing with in PSH sessions has something to do with their buffering semantic (since the reality is that they're buffering XML at its character boundary, or even binary stream; not the rendered characters/colors which it represents). The re-appearance of the prompt may also be a side-effect of this: the dotnet compiler thing can (or could?) be run in an interactive PSH session - that's a lot of LOC including the code which its compiling down to CIL.

Regarding assignment of blame: its kinda like any other software ecosystem - ultimately, the blame rests with the architects who lacked the foresight to predict the machinations of the hordes upon their whiteboard-wonderful creations. Which is exactly why the horrific things you're doing to get this across differentiate the Metasploit team from MSFT :smile:

[sempervictus]

I feel like i should be sending :bouquet: to @smcintyre-r7 and @adfoster-r7 for the insane amount of QA work which this required. Thanks a ton to everyone who helped get this over the finish line. If anyone has time before i do to port the handler fixes to #17600, then i think we can get that one landed as well (probably merits checking if the shell under the SSH connection is a really TTY under the kin).

[smcintyre-r7]

Release Notes

This adds the ability for Metasploit to establish sessions to EC2 instances using Amazon's SSM interface. The result is an interactive shell that does not require the user to transfer a payload to the EC2 instance. For Windows targets, the shell is a a PTY enabled Powershell session that is incompatible with Post modules but supports user interaction.