If whois server is down, try other addresses

[njh]

Hi,

I am having trouble with timeouts when looking up some domains. I am slightly confused about what I might be doing differently/weirdly, because I would expect others to be having the same problem. Looking up using the whois CLI tool works fine.

Code:

#!/usr/bin/env ruby

require 'whois'

p Whois::VERSION

whois = Whois::Client.new(:timeout => 30)
record = whois.lookup('aelius.com')
if record.nil? or record.parts.empty?
  raise "Error: unable to load whois"
end

p record

Result:

"5.0.0"
Traceback (most recent call last):
	12: from test.rb:8:in `<main>'
	11: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/client.rb:91:in `lookup'
	10: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/2.5.0/timeout.rb:108:in `timeout'
	 9: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/client.rb:94:in `block in lookup'
	 8: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/server/adapters/base.rb:112:in `lookup'
	 7: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/server/adapters/base.rb:151:in `buffer_start'
	 6: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/server/adapters/base.rb:113:in `block in lookup'
	 5: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/server/adapters/verisign.rb:33:in `request'
	 4: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/server/adapters/base.rb:183:in `query'
	 3: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/server/socket_handler.rb:38:in `call'
	 2: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/server/socket_handler.rb:55:in `execute'
	 1: from /Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/server/socket_handler.rb:55:in `new'
/Users/njh/.rbenv/versions/2.5.5/lib/ruby/gems/2.5.0/gems/whois-5.0.0/lib/whois/server/socket_handler.rb:55:in `initialize': execution expired (Timeout::Error)

Ruby version:

$ ruby -v
ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-darwin18]

Any advice on how to diagnose the problem further would be appreciated.

Thanks,

nick.

[weppos]

@njh I was able to run the same query. Are you running an high volume of requests? You may have been throttled.

[njh]

No, I am not running a high volume of requests.

My 'production' script looks up 13 domains, with a 1 second pause between each one. But I have this problem when running the one-off test too. On a different IP at a different time.

I am now wondering if it is related to me using IPv6. This is what I found with lsof -ni:

TCP [2001:8b0:ffd5:3:xxxx]:64293->[2406:da00:ff00::34c9:6d7f]:nicname (SYN_SENT)

Do you have IPv6? I guess it is possible there is a problem with their IPv6 network / rate limiting.

[weppos]

Do you have IPv6?

No, I believe I'm currently binding to an IPv4 address.

[weppos]

Closing as not dependent by the library.

[njh]

Just been looking into this and it looks like Amazon's Registrar isn't responding to connections on port 43 when using IPv6. I have reported it on the AWS forum: https://forums.aws.amazon.com/thread.jspa?threadID=317497

If you don't have IPv6, then ruby (or the OS?) won't attempt to connect to the IPv6 address, and happily connect to the working IPv4 address.

Would you accept a PR that switches from TCPSocket to Net::TCPClient?

This would improve IPv4-only connections too, as if a registrar publishes multiple IP addresses in DNS (as most do), it will attempt each one in turn.

[weppos]

Thanks for researching into it @njh! I'm not thrilled about the idea of introducing a dependency to a third party library, especially to circumvent a specific provider issue.

I am interested to explore other options.

Can I ask you how did you find this issue? In Ruby, you can pass a binding IP address to the TPC socket. This library supports it by using bind_host.

c = Whois::Client.new(bind_host: "192.168.1.100")

Would you be able to test by binding the local IPv4 network? I will need to spend some time to determine who's responsible for choosing the IPv6 address over IPv4 when resolving the whois host.

[njh]

I started down this route because my personal domain checking script (that uses this gem) kept failing and timing out. I have IPv6 at home and on my VPS, so it took a while to realise that it was IPv6 related.

It is getting much better, but unfortunately it is not uncommon for some IPv6 networks and services to have periods of downtime. Partly because monitoring is in place for IPv4 but not IPv6.

However this improvement could have benefits for non-IPv6 users too. For example whois.verisign-grs.com has two IPv4 addresses associated with it:

$ host whois.verisign-grs.com
whois.verisign-grs.com has address 192.30.45.30
whois.verisign-grs.com has address 192.34.234.30
whois.verisign-grs.com has IPv6 address 2620:74:20::30
whois.verisign-grs.com has IPv6 address 2620:74:21::30

If one of the servers are down (for example 192.30.45.30), then it won't currently fail over to the other server (192.34.234.30). It will just timeout after attempting the first IP address.

Web browsers have tried to solve this problem using an approach called Happy Eyeballs / RFC8305. However I think something simpler should be possible for whois.

1. Lookup the hostname in DNS
2. Loop through each of the IP addresses returned
3. Attempt to connect to port 43 for each address
4. If successful, send request and return result
5. If unsuccessful, timeout after a few seconds and move on to the next address

It is a shame that ruby doesn't have a internal implementation to do this for you. But I will see if I can write some proof of concept code, to avoid adding an external dependency.

[njh]

This little proof of concept is working for me. It loops through each of the addresses for a hostname and stops when it gets a response:

def whois_request(address, domain)
  result = nil
  Socket.tcp(address, 43, connect_timeout: 5) do |sock|
    sock.write("DOMAIN #{domain}\r\n")
    result = sock.readlines
  end
  result
end

Resolv.each_address('whois.verisign-grs.com') do |address|
  begin
    puts "Trying: #{address}"
    result = whois_request(address, 'aelius.com')
    unless result.nil?
      puts "Success! Read #{result.count} lines."
      break
    end
  rescue Errno::ETIMEDOUT => exp
    puts "Timed out while connecting to #{address}"
  end
end

Note that it uses Socket.tcp() instead of TCPSocket.new because the later doesn't support setting a connect timeout. I think Socket.tcp() was added in Ruby 2.x.

I did try your suggestion of binding the local socket to my IPv4 address and it does work as a workaround (thanks!). I even found that you can bind it to 0.0.0.0 to force IPv4 but not have to hard code a specific address:

whois = Whois::Client.new(:bind_host => "0.0.0.0")

However it feels like a very hacky solution to the problem.

[njh]

I have discovered that Resolv.each_address doesn't respect operating system preferences (for example the configuration in /etc/gai.conf).

Here is a revision that uses Addrinfo.foreach instead, which attempts connections in the correct order:

def whois_request(addrinfo, domain)
  result = nil
  Socket.tcp(addrinfo.ip_address, addrinfo.ip_port, connect_timeout: 5) do |sock|
    sock.write("DOMAIN #{domain}\r\n")
    result = sock.readlines
  end
  result
end

Addrinfo.foreach("whois.verisign-grs.com", "whois", nil, Socket::SOCK_STREAM, Socket::IPPROTO_TCP).each do |addrinfo|
  begin
    puts "Trying: #{addrinfo.inspect}"
    result = whois_request(addrinfo, 'aelius.com')
    unless result.nil?
      puts "Success! Read #{result.count} lines."
      break
    end
  rescue Errno::ETIMEDOUT => exp
    puts "Timed out while connecting to #{addrinfo.ip_address}"
  end
end