metasploit-framework GSoC Project: Implement HTTP-Trace enabled login scanners

This PR implements the HTTP-Trace option for login scanners of the framework. This is achieved by crafting a logging proc and passing it over to Rex::Proto::Http::Client while any HTTP request is sent.

Cause of Bug

Currently the HTTP-Trace feature is implemented in the Msf::Core::Exploit::Remote::HttpClient mixin, hence it is only available for the exploit modules which import its functionality. The Login Scanner library is defined in the Metasploit::Framework::LoginScanner::HTTP mixin and thus is unable to access the HTTP-Trace feature.

Approach

The idea is to implement a proc in Metasploit::Framework::LoginScanner::HTTP library for logging the HTTP requests and responses to the console. The Metasploit::Framework::LoginScanner::HTTP library is the central class to handle the HTTP requests and responses of all the login scanner modules. The Metasploit::Framework::LoginScanner::HTTP library ultimately forwards the HTTP request to Rex::Proto::Http::Client` for transmitting the request to the server and obtaining its response back.

So, the logging proc defined in Metasploit::Framework::LoginScanner::HTTP is passed to Rex::Proto::Http::Client along with the HTTP request, and gets executed on the context of the Rex class. Once the proc is executed, the HTTP requests and responses are logged to the console.

Note: While majority of the Login Scanner libraries (like Metasploit::Framework::LoginScanner::Caidao) access Metasploit::Framework::LoginScanner::HTTP class for sending requests, there are some libraries (like Metasploit::Framework::LoginScanner::Zabbix) which do not access Metasploit::Framework::LoginScanner::HTTP, rather directly transmit the request via Rex::Proto::Http::Client. This PR also takes care of normalizing those login scanner libraries to follow the ideal template of accessing Metasploit::Framework::LoginScanner::HTTP library for sending HTTP requests and receiving responses.

Work Done in the project

Implemented HttpTrace, HttpTraceHeadersOnly, HttpTraceColors options for login scanner modules.
Normalized all login scanner modules to follow one template.
Covered RSpec unit tests for the functionality added.

Before

msf6 > use auxiliary/scanner/http/buffalo_login 
msf6 auxiliary(scanner/http/buffalo_login) > set RHOSTS www.google.com
RHOSTS => www.google.com
msf6 auxiliary(scanner/http/buffalo_login) > set HttpTrace true
HttpTrace => true
msf6 auxiliary(scanner/http/buffalo_login) > set USERPASS_FILE data/wordlists/http_default_userpass.txt
USERPASS_FILE => data/wordlists/http_default_userpass.txt
msf6 auxiliary(scanner/http/buffalo_login) > run

[-] 142.250.193.164:80 - LOGIN FAILED: connect:connect (Incorrect)
[-] 142.250.193.164:80 - LOGIN FAILED: sitecom:sitecom (Incorrect)
[-] 142.250.193.164:80 - LOGIN FAILED: admin:1234 (Incorrect)
[-] 142.250.193.164:80 - LOGIN FAILED: cisco:cisco (Incorrect)
[-] 142.250.193.164:80 - LOGIN FAILED: cisco:sanfran (Incorrect)
[-] 142.250.193.164:80 - LOGIN FAILED: private:private (Incorrect)
[-] 142.250.193.164:80 - LOGIN FAILED: wampp:xampp (Incorrect)
[-] 142.250.193.164:80 - LOGIN FAILED: newuser:wampp (Incorrect)
[-] 142.250.193.164:80 - LOGIN FAILED: xampp-dav-unsecure:ppmax2011  (Incorrect)
[-] 142.250.193.164:80 - LOGIN FAILED: admin:turnkey (Incorrect)
[-] 142.250.193.164:80 - LOGIN FAILED: vagrant:vagrant (Incorrect)
[*] Scanned 1 of 2 hosts (50% complete)
[-] 2404:6800:4007:821::2004:80 - LOGIN FAILED: connect:connect (Unable to Connect)
[-] 2404:6800:4007:821::2004:80 - LOGIN FAILED: sitecom:sitecom (Unable to Connect)
[-] 2404:6800:4007:821::2004:80 - LOGIN FAILED: admin:1234 (Unable to Connect)
[*] Scanned 2 of 2 hosts (100% complete)
[*] Auxiliary module execution completed

After

output

msf6 > use auxiliary/scanner/http/buffalo_login 
msf6 auxiliary(scanner/http/buffalo_login) > set RHOSTS www.google.com
RHOSTS => www.google.com
msf6 auxiliary(scanner/http/buffalo_login) > set USERPASS_FILE data/wordlists/http_default_userpass.txt
USERPASS_FILE => data/wordlists/http_default_userpass.txt
msf6 auxiliary(scanner/http/buffalo_login) > set HttpTrace true
HttpTrace => true
msf6 auxiliary(scanner/http/buffalo_login) > run

####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 51

bufaction=verifyLogin&user=connect&password=connect
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:16 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: connect:connect (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 51

bufaction=verifyLogin&user=sitecom&password=sitecom
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:17 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: sitecom:sitecom (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 46

bufaction=verifyLogin&user=admin&password=1234
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:17 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: admin:1234 (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 47

bufaction=verifyLogin&user=cisco&password=cisco
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:17 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: cisco:cisco (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 49

bufaction=verifyLogin&user=cisco&password=sanfran
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:17 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: cisco:sanfran (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 51

bufaction=verifyLogin&user=private&password=private
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:17 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: private:private (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 47

bufaction=verifyLogin&user=wampp&password=xampp
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:17 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: wampp:xampp (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 49

bufaction=verifyLogin&user=newuser&password=wampp
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:17 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: newuser:wampp (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 67

bufaction=verifyLogin&user=xampp-dav-unsecure&password=ppmax2011%20
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:17 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: xampp-dav-unsecure:ppmax2011  (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 49

bufaction=verifyLogin&user=admin&password=turnkey
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:18 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: admin:turnkey (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12_2_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.81 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Content-Length: 51

bufaction=verifyLogin&user=vagrant&password=vagrant
####################
# Response:
####################
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1571
Date: Mon, 26 Sep 2022 06:21:18 GMT
Connection: close

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/dynamic.pl</code> was not found on this server.  <ins>That’s all we know.</ins>

[-] 142.250.193.164:80 - LOGIN FAILED: vagrant:vagrant (Incorrect)
[*] Scanned 1 of 2 hosts (50% complete)
[-] 2404:6800:4007:821::2004:80 - LOGIN FAILED: connect:connect (Unable to Connect)
[-] 2404:6800:4007:821::2004:80 - LOGIN FAILED: sitecom:sitecom (Unable to Connect)
[-] 2404:6800:4007:821::2004:80 - LOGIN FAILED: admin:1234 (Unable to Connect)
[*] Scanned 2 of 2 hosts (100% complete)
[*] Auxiliary module execution completed

Verification

List the steps needed to make sure this thing works

Start msfconsole
Use any login scanner. (ex - use auxiliary/scanner/http/buffalo_login)
set RHOSTS www.google.com
set USERPASS_FILE data/wordlists/http_default_userpass.txt
set HttpTrace true
run
Verify that the HTTP requests and responses are tracked to the console for every credential tested.
set HttpTraceHeadersOnly true
run
Verify that only the Headers of HTTP requests and responses are tracked to the console for every credential tested.
set HttpTraceColors blu/grn
run
Verify that the requests and responses are tracked to the console in blue and green color respectively.

Acknowledgement

I extend my heartfelt thanks to @gwillcox-r7 @zeroSteiner @adfoster-r7 @jmartin-r7 to guide me throughout the course of the project and for always helping me when I was stuck with the code. This wouldn't be possible without your helping hand :)

Sep 25 '22 10:09 3V3RYONE

@3V3RYONE Sorry for the numerous rebases here, but should now be rebased against latest master, no more merge conflicts, and minus one file that I seem to have done a lot of RuboCop'ing to by accident and haven't quite undone the Rubocop changes to, should be pretty much what you submitted rebased against upstream/master and squashed for neatness.

Sep 25 '22 21:09 gwillcox-r7

@gwillcox-r7 Thanks a lot for resolving the merge conflicts!! Was confused for a bit on how to resolve them (white spaces :sweat_smile: )

Sep 26 '22 04:09 3V3RYONE

@3V3RYONE Above comments will still apply, though I undid the RuboCop changes to lib/msf/core/exploit/remote/http_client.rb so that things are easier to review at the request of my colleagues.

Sep 29 '22 21:09 gwillcox-r7

With this, requests and responses are logged separately and requests are logged before they are sent to the server. This eliminates the above mentioned cases like that of a server not responding. Documentation on the parameters have been added in lib/rex/proto/http/client as well.

Apologies for the late follow up, was stuck up with a bit of personal work. However, I am free now and would be happy to make any further changes if required!

Oct 25 '22 16:10 3V3RYONE

Thank you @3V3RYONE for the great enhancement also for working with and responding to all the comments made by my colleagues! I've reviewed the PR and followed the verification steps and everything seems to be working as intended.

I tested the HttpTraceColor options and they seem to be working fine though I didn't include those tests below because, well, you can see the pretty colors on github.

Module Tests:

buffalo_login

before

msf6 auxiliary(scanner/http/buffalo_login) > set rhosts localhost
rhosts => localhost
msf6 auxiliary(scanner/http/buffalo_login) > set rport 8081
rport => 8081
msf6 auxiliary(scanner/http/buffalo_login) > set USERPASS_FILE data/wordlists/http_default_userpass.txt
USERPASS_FILE => data/wordlists/http_default_userpass.txt
msf6 auxiliary(scanner/http/buffalo_login) > set HttpTrace true
HttpTrace => true
msf6 auxiliary(scanner/http/buffalo_login) > run

[!] No active DB -- Credential data will not be saved!
[-] 0.0.0.1:8081 - LOGIN FAILED: connect:connect (Unable to Connect)
[-] 0.0.0.1:8081 - LOGIN FAILED: sitecom:sitecom (Unable to Connect)
[-] 0.0.0.1:8081 - LOGIN FAILED: admin:1234 (Unable to Connect)
[*] Scanned 1 of 2 hosts (50% complete)
[!] No active DB -- Credential data will not be saved!
[-] 127.0.0.1:8081 - LOGIN FAILED: connect:connect (Incorrect)
[-] 127.0.0.1:8081 - LOGIN FAILED: sitecom:sitecom (Incorrect)
[-] 127.0.0.1:8081 - LOGIN FAILED: admin:1234 (Incorrect)
[-] 127.0.0.1:8081 - LOGIN FAILED: cisco:cisco (Incorrect)
[-] 127.0.0.1:8081 - LOGIN FAILED: cisco:sanfran (Incorrect)
[-] 127.0.0.1:8081 - LOGIN FAILED: private:private (Incorrect)
[-] 127.0.0.1:8081 - LOGIN FAILED: wampp:xampp (Incorrect)
[-] 127.0.0.1:8081 - LOGIN FAILED: newuser:wampp (Incorrect)
[-] 127.0.0.1:8081 - LOGIN FAILED: xampp-dav-unsecure:ppmax2011  (Incorrect)
[-] 127.0.0.1:8081 - LOGIN FAILED: admin:turnkey (Incorrect)
[-] 127.0.0.1:8081 - LOGIN FAILED: vagrant:vagrant (Incorrect)
[*] Scanned 2 of 2 hosts (100% complete)
[*] Auxiliary module execution completed
msf6 auxiliary(scanner/http/buffalo_login) >

after

msf6 auxiliary(scanner/http/buffalo_login) > run rhosts=localhost rport=8081 HttpTrace=true USERPASS_FILE=data/wordlists/http_default_userpass.txt

[!] No active DB -- Credential data will not be saved!
[-] 0.0.0.1:8081 - LOGIN FAILED: connect:connect (Unable to Connect)
[-] 0.0.0.1:8081 - LOGIN FAILED: sitecom:sitecom (Unable to Connect)
[-] 0.0.0.1:8081 - LOGIN FAILED: admin:1234 (Unable to Connect)
[*] Scanned 1 of 2 hosts (50% complete)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: localhost:8081
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 51

bufaction=verifyLogin&user=connect&password=connect
####################
# Response:
####################
HTTP/1.0 501 Unsupported method ('POST')
Server: SimpleHTTP/0.6 Python/3.9.13
Date: Wed, 02 Nov 2022 18:40:36 GMT
Connection: close
Content-Type: text/html;charset=utf-8
Content-Length: 497

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
        "http://www.w3.org/TR/html4/strict.dtd">
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
        <title>Error response</title>
    </head>
    <body>
        <h1>Error response</h1>
        <p>Error code: 501</p>
        <p>Message: Unsupported method ('POST').</p>
        <p>Error code explanation: HTTPStatus.NOT_IMPLEMENTED - Server does not support this operation.</p>
    </body>
</html>

[!] No active DB -- Credential data will not be saved!
[-] 127.0.0.1:8081 - LOGIN FAILED: connect:connect (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: localhost:8081
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 51

bufaction=verifyLogin&user=sitecom&password=sitecom
####################
# Response:
####################
HTTP/1.0 501 Unsupported method ('POST')
Server: SimpleHTTP/0.6 Python/3.9.13
Date: Wed, 02 Nov 2022 18:40:36 GMT
Connection: close
Content-Type: text/html;charset=utf-8
Content-Length: 497

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
        "http://www.w3.org/TR/html4/strict.dtd">
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html;charset=utf-8">
        <title>Error response</title>
    </head>
    <body>
        <h1>Error response</h1>
        <p>Error code: 501</p>
        <p>Message: Unsupported method ('POST').</p>
        <p>Error code explanation: HTTPStatus.NOT_IMPLEMENTED - Server does not support this operation.</p>
    </body>
</html>

HttpTraceHeadersOnly=true

msf6 auxiliary(scanner/http/buffalo_login) > run rhosts=localhost rport=8081 HttpTrace=true HttpTraceHeadersOnly=true USERPASS_FILE=data/wordlists/http_default_userpass.txt

[!] No active DB -- Credential data will not be saved!
[-] 0.0.0.1:8081 - LOGIN FAILED: connect:connect (Unable to Connect)
[-] 0.0.0.1:8081 - LOGIN FAILED: sitecom:sitecom (Unable to Connect)
[-] 0.0.0.1:8081 - LOGIN FAILED: admin:1234 (Unable to Connect)
[*] Scanned 1 of 2 hosts (50% complete)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: localhost:8081
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 51

####################
# Response:
####################
HTTP/1.0 501 Unsupported method ('POST')
Server: SimpleHTTP/0.6 Python/3.9.13
Date: Wed, 02 Nov 2022 18:42:07 GMT
Connection: close
Content-Type: text/html;charset=utf-8
Content-Length: 497


[!] No active DB -- Credential data will not be saved!
[-] 127.0.0.1:8081 - LOGIN FAILED: connect:connect (Incorrect)
####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: localhost:8081
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 51

####################
# Response:
####################
HTTP/1.0 501 Unsupported method ('POST')
Server: SimpleHTTP/0.6 Python/3.9.13
Date: Wed, 02 Nov 2022 18:42:07 GMT
Connection: close
Content-Type: text/html;charset=utf-8
Content-Length: 497


[-] 127.0.0.1:8081 - LOGIN FAILED: sitecom:sitecom (Incorrect)

zabbix_login

before

The zabbix_login module includes `Msf::Exploit::Remote::HttpClient` so HttpTrace worked before, just demonstrating that it also works after these changes.

after

msf6 auxiliary(scanner/http/zabbix_login) > set targeturi /
targeturi => /
msf6 auxiliary(scanner/http/zabbix_login) > run rhosts=172.16.199.131 HttpTrace=true USERPASS_FILE=data/wordlists/http_default_userpass.txt

####################
# Request:
####################
GET / HTTP/1.1
Host: 172.16.199.131
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0


####################
# Response:
####################
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 02 Nov 2022 18:24:54 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
X-Powered-By: PHP/7.4.19
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: zbx_session=eyJzZXNzaW9uaWQiOiI2NWRjOGU1YjQ4NTlkNmQyMGIzYWI2OTQ3MDI5NzdjYiIsInNpZ24iOiI5MTdmNDc2ZDJhMDBlOWM5ZDM1MWZhY2MzYjYzMmU4ZDY3ZWVmMGZmM2E4YjQzOWZkNmFhNzFkMTE1ODhlNTNjIn0%3D; HttpOnly

<!DOCTYPE html>
<html lang="en">
	<head>
		<meta http-equiv="X-UA-Compatible" content="IE=Edge"/>
		<meta charset="utf-8" />
		<meta name="viewport" content="width=device-width, initial-scale=1">
		
	
	truncated due to size	
	</head>
</html>

HttpTraceHeadersOnly

msf6 auxiliary(scanner/http/zabbix_login) > run rhosts=172.16.199.131 HttpTrace=true HttpTraceHeadersOnly=true USERPASS_FILE=data/wordlists/http_default_userpass.txt

####################
# Request:
####################
GET / HTTP/1.1
Host: 172.16.199.131
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0

####################
# Response:
####################
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 02 Nov 2022 18:26:59 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
X-Powered-By: PHP/7.4.19
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Set-Cookie: zbx_session=eyJzZXNzaW9uaWQiOiJjMWI1MjQyNThjMWVkMDE1ZDljMjlhNjkyYmZmZGU0NyIsInNpZ24iOiI5ZTcxNjA0MDgzNTA3ZGViYWY2ZGI5YmZkZjk3MmFmNDAzZDc5NWMzMDc5N2E3ZWE5MjQ1ZmFkMDdmYTQ3ZjIwIn0%3D; HttpOnly


[*] 172.16.199.131:80 - Found Zabbix version 6.2
####################
# Request:
####################
GET /dashboard.php HTTP/1.1
Host: 172.16.199.131
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0

####################
# Response:
####################
HTTP/1.1 404 Not Found
Server: nginx
Date: Wed, 02 Nov 2022 18:26:59 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
X-Powered-By: PHP/7.4.19


####################
# Request:
####################
GET /zabbix.php?action=dashboard.view HTTP/1.1
Host: 172.16.199.131
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0

####################
# Response:
####################
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 02 Nov 2022 18:26:59 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
X-Powered-By: PHP/7.4.19
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Set-Cookie: zbx_session=eyJzZXNzaW9uaWQiOiIzN2ZhMGI2ODE0M2JhNDExMTRhMjRjMWY4MzAxMTYxNSIsInNpZ24iOiI4ZGIxNzg5YzY1MjBkZGM5NzJmOGMwMWNkZjRiNjYxZDE3NGZmM2U0NDA2NzQ4YTdkMDk0ZjAwMTI3Y2QxMDEwIn0%3D; HttpOnly


[*] 172.16.199.131:80 - This Zabbix instance has disabled Guest mode
####################
# Request:
####################
POST /index.php HTTP/1.1
Host: 172.16.199.131
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 63

####################
# Response:
####################
HTTP/1.1 302 Found
Server: nginx
Date: Wed, 02 Nov 2022 18:27:00 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
X-Powered-By: PHP/7.4.19
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Location: zabbix.php?action=dashboard.view
Set-Cookie: zbx_session=eyJzZXNzaW9uaWQiOiJjZjMyNWRhZTU5ZTA1MDg0OTIyZWExNGM5YTA3OTkyNCIsInNpZ24iOiIyZjRhNTNlMTI1MjcyN2M1ZGE5M2E0NDM3MjdhOWEyZmE2MWVlMzk5ZDQ3Njc1NDliNjllZTkzYWQ3NTZiYTA5In0%3D; expires=Fri, 02-Dec-2022 18:27:00 GMT; Max-Age=2592000; HttpOnly


####################
# Request:
####################
GET /profile.php HTTP/1.1
Host: 172.16.199.131
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0
Cookie: zbx_session=eyJzZXNzaW9uaWQiOiJjZjMyNWRhZTU5ZTA1MDg0OTIyZWExNGM5YTA3OTkyNCIsInNpZ24iOiIyZjRhNTNlMTI1MjcyN2M1ZGE5M2E0NDM3MjdhOWEyZmE2MWVlMzk5ZDQ3Njc1NDliNjllZTkzYWQ3NTZiYTA5In0%3D;

####################
# Response:
####################
HTTP/1.1 404 Not Found
Server: nginx
Date: Wed, 02 Nov 2022 18:27:00 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
X-Powered-By: PHP/7.4.19


####################
# Request:
####################
GET /zabbix.php?action=userprofile.edit HTTP/1.1
Host: 172.16.199.131
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0
Cookie: zbx_session=eyJzZXNzaW9uaWQiOiJjZjMyNWRhZTU5ZTA1MDg0OTIyZWExNGM5YTA3OTkyNCIsInNpZ24iOiIyZjRhNTNlMTI1MjcyN2M1ZGE5M2E0NDM3MjdhOWEyZmE2MWVlMzk5ZDQ3Njc1NDliNjllZTkzYWQ3NTZiYTA5In0%3D;

####################
# Response:
####################
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 02 Nov 2022 18:27:00 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Keep-Alive: timeout=20
X-Powered-By: PHP/7.4.19
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Set-Cookie: tab=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0, zbx_session=eyJzZXNzaW9uaWQiOiJjZjMyNWRhZTU5ZTA1MDg0OTIyZWExNGM5YTA3OTkyNCIsInNpZ24iOiIyZjRhNTNlMTI1MjcyN2M1ZGE5M2E0NDM3MjdhOWEyZmE2MWVlMzk5ZDQ3Njc1NDliNjllZTkzYWQ3NTZiYTA5In0%3D; expires=Fri, 02-Dec-2022 18:27:00 GMT; Max-Age=2592000; HttpOnly
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN


[+] 172.16.199.131:80 - Success: 'Admin:zabbix'
[!] No active DB -- Credential data will not be saved!

Spec tests:

All Test

http_spec.rb

msfuser@ubuntu:~/git/metasploit-framework$ rspec spec/lib/metasploit/framework/login_scanner/http_spec.rb

...

Finished in 0.83279 seconds (files took 5.1 seconds to load)
64 examples, 0 failures

chef_webui_spec.rb

msfuser@ubuntu:~/git/metasploit-framework$ rspec spec/lib/metasploit/framework/login_scanner/chef_webui_spec.rb

...

Finished in 0.23059 seconds (files took 4.29 seconds to load)
50 examples, 0 failures

glassfish_spec.rb

msfuser@ubuntu:~/git/metasploit-framework$ rspec spec/lib/metasploit/framework/login_scanner/glassfish_spec.rb

...

Finished in 0.29458 seconds (files took 4.11 seconds to load)
62 examples, 0 failures

zabbix_spec.rb

msfuser@ubuntu:~/git/metasploit-framework$ rspec spec/lib/metasploit/framework/login_scanner/zabbix_spec.rb

...

Finished in 0.24456 seconds (files took 4.23 seconds to load)
50 examples, 0 failures

client_spec.rb

msfuser@ubuntu:~/git/metasploit-framework$ rspec spec/lib/rex/proto/http/client_spec.rb
 
 ...
 
Finished in 0.18923 seconds (files took 4.59 seconds to load)
46 examples, 0 failures, 8 pending

Nov 02 '22 19:11 jheysel-r7

Hello, any update on this?

Nov 13 '22 07:11 3V3RYONE

@3V3RYONE, this is still in testing and review. Testing has gone well, there is still some pending review ongoing to determine if any further adjustment will be need. Sorry for the delay this is definitely still getting worked thru.

Nov 13 '22 18:11 jmartin-tech

Thanks a lot @jmartin-r7, sure thing. Let me know if any further changes are required! Would be happy to work through them :)

Nov 14 '22 06:11 3V3RYONE

Hey @jmartin-r7, just checking in here, any update?

Nov 28 '22 20:11 jheysel-r7

For visibility: I did some testing on a large search space in attempt to expose any possible concurrency issues with access of the http client used in the run. There were no issues with my test:

I scanned 10 HTTP servers with a user password list with 10,000 entries.

msf6 auxiliary(scanner/http/buffalo_login) > run HttpTrace=true HttpTraceHeadersOnly=true USERPASS_FILE=passfile.txt rport=8081 rhosts=172.16.199.246 172.16.199.164 172.16.199.158 172.16.199.132 172.16.199.128 172.16.199.133 172.16.199.130 172.16.199.242 172.16.199.131 localhost

...


####################
# Request:
####################
POST /dynamic.pl HTTP/1.1
Host: localhost:8081
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 12.2; rv:97.0) Gecko/20100101 Firefox/97.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 58

####################
# Response:
####################
HTTP/1.0 501 Unsupported method ('POST')
Server: SimpleHTTP/0.6 Python/3.9.13
Date: Wed, 02 Nov 2022 21:19:14 GMT
Connection: close
Content-Type: text/html;charset=utf-8
Content-Length: 497


[-] 127.0.0.1:8081 - LOGIN FAILED: admin:password10000: (Incorrect)
[*] Scanned 11 of 11 hosts (100% complete)
[*] Auxiliary module execution completed

userpass file generated with: for x in {1..10000}; do echo "admin:password$x" >> passfile.txt; done

Dec 05 '22 14:12 jheysel-r7

@3V3RYONE Any update on addressing the comments on this PR? We'd like to try get this in before the next GSOC starts and are here on hand to provide advice should you need guidance on anything. There have been quite a lot of updates to the codebase so we'll likely also need to rebase this against the latest upstream changes and make sure these changes don't break anything.

6.3 will also be going out shortly so it may also be worthwhile to double check things after that gets released.

Jan 23 '23 21:01 gwillcox-r7

Hello, sincere apologies for my late follow up. I'm surely resuming work on this by the current week. I'm back with contributing to the framework with other issues as well, and will surely get this one landed before next GSoC.

Jan 23 '23 21:01 3V3RYONE

@3V3RYONE Let me know if you need any help with this :+1:

Feb 22 '23 14:02 adfoster-r7

Going to attic this so long as this needs more work to get to the point where we can land this and I haven't seen any significant updates on this for a while. When the issues above have been addressed and this has been rebased we can unattic this and give this another shot.

Note that you can still submit updates to this PR whilst its attic'd though, so don't feel like this is a blocker to you still working on this as we can reopen the PR at any time once the issues have been fixed.

Mar 06 '23 16:03 gwillcox-r7

Thanks for your contribution to Metasploit Framework! We've looked at this pull request, and we agree that it seems like a good addition to Metasploit, but it looks like it is not quite ready to land. We've labeled it attic and closed it for now.

What does this generally mean? It could be one or more of several things:

It doesn't look like there has been any activity on this pull request in a while
We may not have the proper access or equipment to test this pull request, or the contributor doesn't have time to work on it right now.
Sometimes the implementation isn't quite right and a different approach is necessary.

We would love to land this pull request when it's ready. If you have a chance to address all comments, we would be happy to reopen and discuss how to merge this!

Mar 06 '23 16:03 github-actions[bot]

@3V3RYONE has been reaching out via slack; Will remove the attic label for now :+1:

Mar 06 '23 17:03 adfoster-r7

Implemented the requested changes and covered them with proper unit tests. Please feel free to let me know for any other changes to be done, thanks! :)

Mar 14 '23 19:03 3V3RYONE

I wrote a quick resource script to run through all the modules to make sure there's no errors:

resource.rc

<ruby>

scanners = %w[
  auxiliary/scanner/http/ipboard_login
  auxiliary/scanner/http/cisco_firepower_login
  auxiliary/scanner/http/mybook_live_login
  auxiliary/scanner/http/hp_sys_mgmt_login.rb
  auxiliary/scanner/http/advantech_webaccess_login
  auxiliary/scanner/http/wordpress_xmlrpc_login
  auxiliary/scanner/http/wordpress_multicall_creds
  auxiliary/scanner/http/gitlab_login
  auxiliary/scanner/http/syncovery_linux_login
  auxiliary/scanner/http/phpmyadmin_login
  auxiliary/scanner/http/symantec_web_gateway_login
  auxiliary/scanner/http/softing_sis_login
  auxiliary/scanner/http/directadmin_login
  auxiliary/scanner/http/zabbix_login
  auxiliary/scanner/http/manageengine_desktop_central_login
  auxiliary/scanner/http/bavision_cam_login
  auxiliary/scanner/http/jenkins_login
  auxiliary/scanner/http/octopusdeploy_login
  auxiliary/scanner/http/buffalo_login
  auxiliary/scanner/http/chef_webui_login
  auxiliary/scanner/nessus/nessus_rest_login
  auxiliary/scanner/http/caidao_bruteforce_login
  auxiliary/scanner/http/axis_login
  auxiliary/scanner/http/jupyter_login
  auxiliary/scanner/http/wowza_streaming_engine_manager_login
  auxiliary/scanner/winrm/winrm_login
  auxiliary/scanner/http/glassfish_login
]

scanners.each do |scanner|
  print_good("==============================================")
  print_good("Running next scanner #{scanner.inspect}")
  print_good("==============================================")
  run_single("use #{scanner}")
  run_single("run ssl=false rhost=127.0.0.1 rport=8000 username=my_user password=p4$$w0rd HttpTrace=true")
  print_line
  print_line
end

</ruby>

1) nessus_rest_login

It looks like the the scanner/nessus/nessus_rest_login module crashed before the nil check for framework_module was added:

msf6 auxiliary(scanner/nessus/nessus_rest_login) > run rhost=192.168.123.13 username=administrator password=p4$$w0rd11 httptrace=true

[-] Auxiliary failed: Rex::RuntimeError Incompatible logger
[-] Call stack:
[-]   /Users/user/Documents/code/metasploit-framework/lib/rex/proto/http/http_logger_subscriber.rb:13:in `initialize'
[-]   /Users/user/Documents/code/metasploit-framework/lib/metasploit/framework/login_scanner/http.rb:337:in `new'
[-]   /Users/user/Documents/code/metasploit-framework/lib/metasploit/framework/login_scanner/http.rb:337:in `create_client'
[-]   /Users/user/Documents/code/metasploit-framework/lib/metasploit/framework/login_scanner/http.rb:243:in `block in send_request'
[-]   /Users/user/Documents/code/metasploit-framework/lib/metasploit/framework/login_scanner/http.rb:243:in `fetch'
[-]   /Users/user/Documents/code/metasploit-framework/lib/metasploit/framework/login_scanner/http.rb:243:in `send_request'
[-]   /Users/user/Documents/code/metasploit-framework/lib/metasploit/framework/login_scanner/nessus.rb:21:in `check_setup'
[-]   /Users/user/Documents/code/metasploit-framework/modules/auxiliary/scanner/nessus/nessus_rest_login.rb:130:in `run_host'
[-]   /Users/user/Documents/code/metasploit-framework/lib/msf/core/auxiliary/scanner.rb:124:in `block (2 levels) in run'
[-]   /Users/user/Documents/code/metasploit-framework/lib/msf/core/thread_manager.rb:105:in `block in spawn'
[-]   /Users/user/.rvm/gems/ruby-3.0.5@metasploit-framework/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
[*] Auxiliary module execution completed

I believe the module will need to be updated to correctly pass in the framework_module for logging to work :+1:

Mar 15 '23 23:03 adfoster-r7

Just to leave a note with the latest details:

We'll want to fix the nessus scanner bug - https://github.com/rapid7/metasploit-framework/pull/17060#issuecomment-1470995543
Change when the request/response gets logged, so we can see encrypted winrm kerberos/ntlm soap traffic https://github.com/rapid7/metasploit-framework/pull/17060/files#r1137957935
swapping to always sending with cgi
Updating data%5BLogin%5D%5Bowner_name%5D' => 'admin', to 'data[Login][owner_name]' => 'admin', as well as the password field, as the library will do that encoding for us https://github.com/rapid7/metasploit-framework/pull/17060/files#r1136097095
Add yard docs to the methods

Apr 03 '23 10:04 adfoster-r7

I ran through Jack's testing steps again, and the previously mentioned issues - and all seems to be well :+1:

The functionality is pretty awesome, thanks! 🎉

May 11 '23 14:05 adfoster-r7

Thank you so much for the constant help and guidance during the project! Finally, it's merged! :tada:

May 11 '23 14:05 3V3RYONE

Release Notes

Updates the HTTP scanner modules with the functionality to log both HTTP requests and responses. This functionality can be enabled with set HTTPTrace true. This functionality is useful for debugging modules. In scenarios where the traffic is encrypted, for instance with WinRM, the logged values will be unencrypted.

May 11 '23 14:05 adfoster-r7