Implement Caching DNS Resolver in Rex

[sempervictus]

trafficstars

Rex::Proto::DNS::Resolver is currently unable to approximate the host OS' native resolver because:

1. It cannot cache responses and has to go out to its defined NS' each time to query for the answers,
2. Because it is not aware of the system's hostsfile entries which can result in leaks/mis-targeted execution, and a bunch of other unpleasantly nuanced problems.

Address the concern by:

1. Creating a descendant CachedResolver class from Rex::Proto::DNS::Resolver, with a #send method override which performs cache query and population.
2. Moving the Cache class up one namespace to Rex::Proto::DNS and updating the server accordingly.
3. Fixing the MATCH_HOSTNAME regex in Rex::Proto::DNS::Constants to allow a short-name (vs FQDN) and creating a relevant MATCH_FQDN.

TODO:

1. Deal with adding search domains from the system to short-name queries and records; if we decide this is a good idea (potential for leaks).
2. Look at performance optimization for multiple concurrent queries via singleton/refcounted/other optimized concurrent access patters.

Testing:

1. Pry-level tests of the objects edited/created in this PR. Needs some runtime testing to QA.

Verification

List the steps needed to make sure this thing works

• [x] Start msfconsole
• [x] pry
• [x] resolver = Rex::Proto::DNS::CachedResolver.new
• [x] resolver.nameserver = '8.8.4.4'
• [x] Verify local hostsfile entries are shown in a call to resolver.cache
• [x] Verify resolver.query('google.com') does not query the OS' own resolver (such as those in /etc/resov.conf)

[gwillcox-r7]

Marking this as delayed @sempervictus due to still waiting on an update on this, but feel free to come back to this when you have time.

[sempervictus]

Thanks for the pinging - lots of stuff in-flight, lost track of this one.

[gwillcox-r7]

@sempervictus Got a few other comments open, if you could take a look at them and add in the extra information or explain the regexes etc being used a bit more, should hopefully be able to wrap this up soon 👍

[gwillcox-r7]

Testing results:

Msfconsole:

msf6 payload(windows/x64/meterpreter/reverse_tcp) > pry
[*] Starting Pry shell...
[*] You are in the "payload/windows/x64/meterpreter/reverse_tcp" module object

[1] pry(#<#<Class:0x00007f117cf42650>>)> resolver = Rex::Proto::DNS::CachedResolver.new
=> ;; RESOLVER state:
;; config_file: /dev/null 	log_file: /dev/null 	
;; port: 53 	searchlist: [] 	
;; nameservers: ["127.0.0.1"] 	domain: "" 	
;; source_port: 0 	source_address: 0.0.0.0 	
;; retry_interval: 5 	retry_number: 4 	
;; recursive: true 	defname: true 	
;; dns_search: true 	use_tcp: false 	
;; ignore_truncated: false 	packet_size: 512 	
;; tcp_timeout: 30 	udp_timeout: 30 	
;; context:  	comm:  	
;; 
[2] pry(#<#<Class:0x00007f117cf42650>>)> resolver.nameserver = '8.8.4.4'
=> "8.8.4.4"
[3] pry(#<#<Class:0x00007f117cf42650>>)> resolver.cache
=> #<Rex::Proto::DNS::Cache:0x00005573a51404b0
 @lock=#<Thread::Mutex:0x00005573a51402a8>,
 @monitor_thread=
  #<Thread:0x00005573a50e50d8 /home/gwillcox/.rvm/gems/ruby-3.0.2@metasploit-framework/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 sleep>,
 @records={}>
[4] pry(#<#<Class:0x00007f117cf42650>>)> resolver.query('google.com')
=> #<Dnsruby::Message:0x00007f117d82a830
 @additional=[],
 @answer=
  [#<Dnsruby::RR::IN::A:0x00007f117d740ff0
    @address=#<Dnsruby::IPv4 142.251.32.206>,
    @klass=IN,
    @name=#<Dnsruby::Name: google.com.>,
    @rdata=#<Dnsruby::IPv4 142.251.32.206>,
    @ttl=272,
    @type=A>],
 @answerfrom=nil,
 @answerip=nil,
 @authority=[],
 @cached=false,
 @do_caching=true,
 @do_validation=true,
 @header=
  #<Dnsruby::Header:0x00007f117d828df0
   @aa=false,
   @ad=false,
   @ancount=0,
   @arcount=0,
   @cd=false,
   @id=15014,
   @nscount=0,
   @opcode=Query,
   @qdcount=1,
   @qr=true,
   @ra=false,
   @rcode=NOERROR,
   @rd=true,
   @tc=false>,
 @question=
  [#<Dnsruby::Question:0x00007f117d7fb6c0
    @qclass=IN,
    @qname=#<Dnsruby::Name: google.com.>,
    @qtype=A>],
 @security_error=nil,
 @security_level=UNCHECKED,
 @send_raw=false,
 @signing=false,
 @tsigkey=nil,
 @tsigstate=:Unsigned>
[5] pry(#<#<Class:0x00007f117cf42650>>)> 

/etc/hosts file

 ~/git/metasploit-framework │ feature/caching_dns_resolver ⇡1 ?29  cat /etc/hosts      
127.0.0.1	localhost
127.0.1.1	gwillcox-Virtual-Machine
10.10.10.216    git.laboratory.htb

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
 ~/git/metasploit-framework │ feature/caching_dns_resolver ⇡1 ?29 

Packet capture

 ~/git/metasploit-framework │ feature/caching_dns_resolver ⇡1 ?29  sudo tshark
Running as user "root" and group "root". This could be dangerous.
Capturing on 'eth0'
 ** (tshark:224199) 13:42:16.890004 [Main MESSAGE] -- Capture started.
 ** (tshark:224199) 13:42:16.890085 [Main MESSAGE] -- File: "/tmp/wireshark_eth04VUTL1.pcapng"
    1 0.000000000  172.28.16.1 → 239.255.255.250 SSDP 216 M-SEARCH * HTTP/1.1 
    2 1.014264641  172.28.16.1 → 239.255.255.250 SSDP 216 M-SEARCH * HTTP/1.1 
    3 1.824140108 172.28.29.166 → 8.8.4.4      DNS 70 Standard query 0x3aa6 A google.com
    4 1.843710482      8.8.4.4 → 172.28.29.166 DNS 86 Standard query response 0x3aa6 A google.com A 142.251.32.206
    5 2.024392267  172.28.16.1 → 239.255.255.250 SSDP 216 M-SEARCH * HTTP/1.1 
    6 3.039396711  172.28.16.1 → 239.255.255.250 SSDP 216 M-SEARCH * HTTP/1.1 
    7 6.930968648 Microsof_a6:01:61 → Microsof_a6:01:00 ARP 42 Who has 172.28.16.1? Tell 172.28.29.166
    8 6.931165649 Microsof_a6:01:00 → Microsof_a6:01:61 ARP 42 172.28.16.1 is at 00:15:5d:a6:01:00
    9 23.014557755 172.28.28.140 → 239.255.255.250 SSDP 179 M-SEARCH * HTTP/1.1 
   10 26.004912179 172.28.28.140 → 239.255.255.250 SSDP 179 M-SEARCH * HTTP/1.1 
   11 29.012514468 172.28.28.140 → 239.255.255.250 SSDP 179 M-SEARCH * HTTP/1.1 
   12 30.110634326 35.160.51.228 → 172.28.29.166 TLSv1.2 97 Application Data
   13 30.150972279 172.28.29.166 → 35.160.51.228 TCP 66 50574 → 443 [ACK] Seq=1 Ack=32 Win=501 Len=0 TSval=479585677 TSecr=3049200005
   14 30.345122614 172.28.29.166 → 35.160.51.228 TLSv1.2 101 Application Data
   15 30.416859885 35.160.51.228 → 172.28.29.166 TCP 66 443 → 50574 [ACK] Seq=32 Ack=36 Win=119 Len=0 TSval=3049200311 TSecr=479585871
^C15 packets captured
 ~/git/metasploit-framework │ feature/caching_dns_resolver ⇡1 ?29

And here is the updated cache after the above query:

[5] pry(#<#<Class:0x00007f117cf42650>>)> resolver.cache
=> #<Rex::Proto::DNS::Cache:0x00005573a51404b0
 @lock=#<Thread::Mutex:0x00005573a51402a8>,
 @monitor_thread=
  #<Thread:0x00005573a50e50d8 /home/gwillcox/.rvm/gems/ruby-3.0.2@metasploit-framework/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:471 sleep>,
 @records=
  {#<Dnsruby::RR::IN::A:0x00007f117d740ff0
    @address=#<Dnsruby::IPv4 142.251.32.206>,
    @klass=IN,
    @name=#<Dnsruby::Name: google.com.>,
    @rdata=#<Dnsruby::IPv4 142.251.32.206>,
    @ttl=272,
    @type=A>=>1652208413}>
[6] pry(#<#<Class:0x00007f117cf42650>>)> 

[gwillcox-r7]

@sempervictus Looks like the majority of this is working but the Verify local hostsfile entries are shown in a call to resolver.cache verification step is failing as I'm not seeing my git.labratory.htb entry in the resolver cache.

[gwillcox-r7]

@sempervictus Just checking in on this and pinging you as a reminder, but there are a few comments here that I'd appreciate you taking a look into when you have the time.

[gwillcox-r7]

Due to no response and no updates on this work in several months now, I'm going to attic this.

[github-actions[bot]]

Thanks for your contribution to Metasploit Framework! We've looked at this pull request, and we agree that it seems like a good addition to Metasploit, but it looks like it is not quite ready to land. We've labeled it attic and closed it for now.

What does this generally mean? It could be one or more of several things:

• It doesn't look like there has been any activity on this pull request in a while
• We may not have the proper access or equipment to test this pull request, or the contributor doesn't have time to work on it right now.
• Sometimes the implementation isn't quite right and a different approach is necessary.

We would love to land this pull request when it's ready. If you have a chance to address all comments, we would be happy to reopen and discuss how to merge this!

[sempervictus]

@gwillcox-r7 - my /etc/hosts file is processed-in correctly with the CachedResolver returning the correct responses (static and with appropriate TTLs).

[53] pry(#<Msf::Framework>)> `grep svl-zen01 /etc/hosts`
=> "192.168.121.132 svl-zen01\n"
[54] pry(#<Msf::Framework>)> Rex::Socket.class_variable_get(:@@resolver).cache.find('svl-zen01')
=> [#<Dnsruby::RR::IN::A:0x000072ba479545d8 @address=#<Dnsruby::IPv4 192.168.121.132>, @klass=IN, @name=#<Dnsruby::Name: svl-zen01>, @ttl=0, @type=A>]
[55] pry(#<Msf::Framework>)> Rex::Socket.class_variable_get(:@@resolver).cache.find('google.com')
=> []
[56] pry(#<Msf::Framework>)> Rex::Socket.class_variable_get(:@@resolver).send('google.com')
=> #<Dnsruby::Message:0x000072ba44356b98
 @additional=[],
 @answer=[#<Dnsruby::RR::IN::A:0x000072ba440cec18 @address=#<Dnsruby::IPv4 142.250.80.14>, @klass=IN, @name=#<Dnsruby::Name: google.com.>, @rdata=#<Dnsruby::IPv4 142.250.80.14>, @ttl=300, @type=A>],
 @answerfrom=nil,
 @answerip=nil,
 @authority=[],
 @cached=false,
 @do_caching=true,
 @do_validation=true,
 @header=#<Dnsruby::Header:0x000072ba44356468 @aa=false, @ad=false, @ancount=0, @arcount=0, @cd=false, @id=2786, @nscount=0, @opcode=Query, @qdcount=1, @qr=true, @ra=false, @rcode=NOERROR, @rd=true, @tc=false>,
 @question=[#<Dnsruby::Question:0x000072ba44355810 @qclass=IN, @qname=#<Dnsruby::Name: google.com.>, @qtype=A>],
 @security_error=nil,
 @security_level=UNCHECKED,
 @send_raw=false,
 @signing=false,
 @tsigkey=nil,
 @tsigstate=:Unsigned>
[57] pry(#<Msf::Framework>)> Rex::Socket.class_variable_get(:@@resolver).cache.find('google.com')
=> [#<Dnsruby::RR::IN::A:0x000072ba440cec18 @address=#<Dnsruby::IPv4 142.250.80.14>, @klass=IN, @name=#<Dnsruby::Name: google.com.>, @rdata=#<Dnsruby::IPv4 142.250.80.14>, @ttl=300, @type=A>]

[gwillcox-r7]

@sempervictus Thanks for the update, will take a look into this again tomorrow once 6.3 updates roll out and let you know if I need anything further to land this; hopefully should be able to unblock this so it can help push https://github.com/rapid7/rex-socket/pull/43 along. Sorry for closing this early; assumed it had been abandoned.

[sempervictus]

Thank you sir - hopefully it works the same way for you and i dont have to go hunting fork-ghosts in the git logs :smile:

[gwillcox-r7]

@sempervictus I'm not seeing an update to this code since I last tried to test this so I presume the issue would still be the same as the last comment I left?

[sempervictus]

@gwillcox-r7 - sorry, not tracking: what is the concern? Far as functional testing goes, i pasted my output above showing hostfile-based resolution.

[gwillcox-r7]

Rebasing this on top of latest updates since there has been quite a few changes since this was last updated.

[gwillcox-r7]

Here are my results when doing this, don't know if the tab character is causing issues, but not quite getting what you showed above. This is after applying the rebase to make sure we are applying this to the latest version of Metasploit aka the recent 6.3 release.

[14] pry(#<Msf::Framework>)> `grep gwillcox-Virtual-Machine /etc/hosts`
=> "127.0.1.1\tgwillcox-Virtual-Machine\n"
[15] pry(#<Msf::Framework>)> resolver.cache.find('gwillcox-Virtual-Machine')
=> []
[16] pry(#<Msf::Framework>)> resolver.cache
=> #<Rex::Proto::DNS::Cache:0x000055f429141630
 @lock=#<Thread::Mutex:0x000055f429141590>,
 @monitor_thread=#<Thread:0x000055f42913a740 /home/gwillcox/.rbenv/versions/3.0.5/lib/ruby/gems/3.0.0/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:471 sleep>,
 @records={}>
[17] pry(#<Msf::Framework>)> resolver.cache.find('google.com')
=> []
[18] pry(#<Msf::Framework>)> resolver.cache.find('www.google.com')
=> []
[19] pry(#<Msf::Framework>)> 

[gwillcox-r7]

Update: Nope looks like even without the tabs this isn't making a difference, still can't find the entries from /etc/hosts:

[3] pry(#<Msf::Framework>)> resolver = Rex::Proto::DNS::CachedResolver.new
=> ;; RESOLVER state:
;; config_file: /dev/null       log_file: /dev/null 
;; port: 53     searchlist: [] 
;; nameservers: ["127.0.0.1"]   domain: "" 
;; source_port: 0       source_address: 0.0.0.0 
;; retry_interval: 5    retry_number: 4 
;; recursive: true      defname: true 
;; dns_search: true     use_tcp: false 
;; ignore_truncated: false      packet_size: 512 
;; tcp_timeout: 30      udp_timeout: 30 
;; context:     comm:  
;; 
[4] pry(#<Msf::Framework>)> resolver.cache.find('gwillcox-Virtual-Machine')
=> []
[5] pry(#<Msf::Framework>)> `grep gwillcox-Virtual-Machine /etc/hosts`
=> "127.0.1.1 gwillcox-Virtual-Machine\n"
[6] pry(#<Msf::Framework>)> 

[sempervictus]

@gwillcox-r7: thank you. That hostsfile ingestion isn't exactly optional - its part of the #initialize method for the CachedResolver:

    def initialize(config = {})
      super(config)
      self.cache = Rex::Proto::DNS::Cache.new
      # Read hostsfile into cache
      hf = Rex::Compat.is_windows ? '%WINDIR%/system32/drivers/etc/hosts' : '/etc/hosts'
      entries = File.read(hf).lines.map(&:strip).select do |entry|
...

so i'd expect something in the framework log if it fails... Could you please manually run through the cache creation/population routine in IRB and step through the read-in? Might be something platform-specific. Also, if you inspect the .cache on your resolver, is there anything in it?

[gwillcox-r7]

@sempervictus Sure give me one sec, I'll run through that now, and will double check the log file for Metasploit.

Edit: Just realized I can debug this with debug.gem in VSCode, give me a sec and will set that up.

[gwillcox-r7]

Okay so debugged this down to the line you mentioned and entries at that point is [["127.0.0.1", "localhost"], ["127.0.1.1", "gwillcox-Virtual-Machine"], ["10.10.10.216", "git.laboratory.htb"]].

Looks like from there some of the code skips over anything starting with 127. so explains the first two entries not showing up. As for the other part looks like we hit the line self.cache.add_static(hostname, ent.first) unless MATCH_HOSTNAME.match hostname with the ent value equalling ["10.10.10.216"] and the hostname value equaling git.laboratory.htb. This results in self.cache.add_static being called but I don't see anything being updated as a result which suggests an error is occurring in that function but we aren't throwing any exceptions.

[gwillcox-r7]

Okay so looks like that unless MATCH_HOSTNAME.match hostname code is preventing the cache call from ever occurring. When I call it manually without that check it works fine. I'm guessing this should be changed to an if statement.

[gwillcox-r7]

Okay with that change this now works:

[1] pry(#<Msf::Framework>)> resolver = Rex::Proto::DNS::CachedResolver.new
=> ;; RESOLVER state:
;; config_file: /dev/null       log_file: /dev/null 
;; port: 53     searchlist: [] 
;; nameservers: ["127.0.0.1"]   domain: "" 
;; source_port: 0       source_address: 0.0.0.0 
;; retry_interval: 5    retry_number: 4 
;; recursive: true      defname: true 
;; dns_search: true     use_tcp: false 
;; ignore_truncated: false      packet_size: 512 
;; tcp_timeout: 30      udp_timeout: 30 
;; context:     comm:  
;; 
[2] pry(#<Msf::Framework>)> resolver.cache
=> #<Rex::Proto::DNS::Cache:0x00007faf4213c948
 @lock=#<Thread::Mutex:0x00007faf4213c8a8>,
 @monitor_thread=
  #<Thread:0x000055dc6a08a240 /home/gwillcox/.rbenv/versions/3.0.5/lib/ruby/gems/3.0.0/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:471 sleep>,
 @records=
  {#<Dnsruby::RR::IN::A:0x000055dc6a090280
    @address=#<Dnsruby::IPv4 10.10.10.216>,
    @klass=IN,
    @name=#<Dnsruby::Name: git.laboratory.htb>,
    @ttl=0,
    @type=A>=>0}>
[3] pry(#<Msf::Framework>)> resolver.cache.find('git.laboratory.htb')
=> [#<Dnsruby::RR::IN::A:0x000055dc6a090280
  @address=#<Dnsruby::IPv4 10.10.10.216>,
  @klass=IN,
  @name=#<Dnsruby::Name: git.laboratory.htb>,
  @ttl=0,
  @type=A>]
[4] pry(#<Msf::Framework>)> resolver.nameserver = '8.8.4.4'
=> "8.8.4.4"
[5] pry(#<Msf::Framework>)> resolver.cache
=> #<Rex::Proto::DNS::Cache:0x00007faf4213c948
 @lock=#<Thread::Mutex:0x00007faf4213c8a8>,
 @monitor_thread=
  #<Thread:0x000055dc6a08a240 /home/gwillcox/.rbenv/versions/3.0.5/lib/ruby/gems/3.0.0/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:471 sleep>,
 @records=
  {#<Dnsruby::RR::IN::A:0x000055dc6a090280
    @address=#<Dnsruby::IPv4 10.10.10.216>,
    @klass=IN,
    @name=#<Dnsruby::Name: git.laboratory.htb>,
    @ttl=0,
    @type=A>=>0}>
[6] pry(#<Msf::Framework>)> resolver.query('google.com')
=> #<Dnsruby::Message:0x000055dc6968ab00
 @additional=[],
 @answer=
  [#<Dnsruby::RR::IN::A:0x000055dc69662da8
    @address=#<Dnsruby::IPv4 142.251.33.14>,
    @klass=IN,
    @name=#<Dnsruby::Name: google.com.>,
    @rdata=#<Dnsruby::IPv4 142.251.33.14>,
    @ttl=196,
    @type=A>],
 @answerfrom=nil,
 @answerip=nil,
 @authority=[],
 @cached=false,
[7] pry(#<Msf::Framework>)> resolver.query('google.com')
=> #<Dnsruby::Message:0x000055dc69470f40
 @additional=[],
 @answer=
  [#<Dnsruby::RR::IN::A:0x000055dc69662da8
    @address=#<Dnsruby::IPv4 142.251.33.14>,
    @klass=IN,
    @name=#<Dnsruby::Name: google.com.>,
    @rdata=#<Dnsruby::IPv4 142.251.33.14>,
    @ttl=196,
    @type=A>],
 @answerfrom=nil,
 @answerip=nil,
 @authority=[],
 @cached=false,
[8] pry(#<Msf::Framework>)> 

[gwillcox-r7]

Will push up the change now. Can also confirm there are no extra queries after we cache the results.

[sempervictus]

Thanks for fixing that, i'm a bit confused as to what was changed as github did something funky to the git history.

[gwillcox-r7]

Thanks for fixing that, i'm a bit confused as to what was changed as github did something funky to the git history.

Ah I rebased this against latest updates so that may have altered something, though I thought I downloaded latest code before the rebase and kept all your commits so IDK what happened 🤔

[sempervictus]

Ah, you can PR to my branch from which this PR grows - easiest way to fix this stuff AFAIK. I can force-push last state (i think i have it) if you want to try that and then we'll actually have a proper commit record of your fix.

[gwillcox-r7]

@sempervictus Sounds good, feel free to go ahead with that and I can do a PR instead.

[gwillcox-r7]

PR raised at https://github.com/sempervictus/metasploit-framework/pull/35 to implement the fix.

[gwillcox-r7]

After discussing this with Spencer I think this is a nice addition however given the existing DNS base of our code is iffy from what I've been told, this may take a bit longer to review than I was initially expecting. Going to pivot off of this for the time being until I can review this a bit more. In the meantime if you are able to provide some live testing examples, that would greatly speed things up on our end.

At the request of Spencer, CCing in @jmartin-r7 here: Do you think this will have any impact on Pro?

[sempervictus]

I can fill in the history on that subtree - its pretty much all my fault :smile:.

The DNS work being pushed upstream has existed in my fork in different forms/places for many years (it will soon be able to get a learner's permit), starting out as a Rex::Socket override for Net::DNS' use of ::Sockets - predaing DnsRuby's existence. Over the years, i fleshed out the Resolver, wrote the Server, Cache, etc while DnsRuby came to life causing some modules/libs to use one and some to use another - a proper :goat: ... which i tried to address by making the Rex::Proto::Dns pieces adaptable to the various DNS libs (it should handle consumers passing requests of either type). The code isn't so much "shaky" as it is procedurally complicated (starting with the #send nonsense inherited from the first commit from Net::DNS).

If framework can agree to not use external DNS libs above the Rex namespace, i can cut the Rex::Proto::DNS code down considerably in terms of simplicity and overhaul the Resolver to use DnsRuby entirely (non-trivial, but do-able). Past that, its all pretty straightforward rex-proto code.

Lastly, stability-wise - i use the full DNS-intercept stack day to day, plumb all of my DNS over TOR as required, and other than the TOR-related time-outs on occasion, it even handles ENUM_BRT at 32 threads correctly (anecdotally compared with results from other DNS enum tools on the same wordlist without TOR).

[gwillcox-r7]

@sempervictus I think @smcintyre-r7 and I were discussing moving things down to just one DNS library and that is an interest of ours though the main concern would be potential side effects if such a switch were to be made, but we still think it would be a good idea to do.

As for not using external DNS libs above the Rex namespace I'm afraid I don't have a good answer for you on that myself, but perhaps Spencer or Alan might have a better answer there for you.