Control SNI hostname used?

[rchady]

I've been searching for a HTTP library that allows me to control the name used for SNI. The use case is I'm connecting to individual nodes in a cluster to test over SS. This means I need to connect to 1 hostname, but send a different one for SNI. Is this something you would be willing to support?

[ixti]

Although it's doable to allow pass different SNI, I don't see how this is a normal use case. From what I understand you want to do something like this?:

# pseudo-code - don't expect it to work yet
HTTP.get("https://a.example.com/foo/bar", :sni => "b.example.com")

[ixti]

In any case, from my understanding, @httprb/core it's pretty doable to allow pass SNI hostname to be used instead of req.uri.host.

[rchady]

Yes. In my case I need to connect to hostA in the cluster over SSL, but check virtual host hostB. In order to do that, I need to be able to control the name used in SNI.

Thanks for the quick response!

[ixti]

Ok. So here's the tricky part: this should also mess with Host header to be same as SNI. In other words, it seems like the API should in fact look more like:

HTTP.connect("a.example.com").get("https://b.example.com/foo/bar")

In other words we need to open SSL socket to a.example.com, but HTTP will be sending request as for b.example.com. Right now you can achieve that by monkey-patching request objects:

# this is fully working example

req = HTTP.build_request(:get, "https://b.example.com")
req.define_singleton_method(:socket_host) { "a.example.com" }
HTTP.perform(req)

The above is just an example, and you can make your life easier with following snippet:

class HTTPSTester < HTTP::Client
  class Request < DelegateClass(HTTP::Request)
    attr_reader :socket_host

    def initialize(socket_host, *args)
      @socket_host = socket_host
      super(*args)
    end
  end

  def initialize(socket_host:, **default_options)
    @socket_host = socket_host
    super(default_options)
  end

  def build_request(*)
    Request.new(@socket_host, super)
  end
end

---

I tend to think that this is actually an edge use case and probably should not be included into this client's core, but I might be wrong. Please, @httprb/core comment what you think.

[tarcieri]

Perhaps we could add an explicit host option to Http::Options which would:

• Set SNI
• Set the Host header
• Set the hostname to use when verifying the TLS certificate

[ixti]

@tarcieri I was thinking about that, but that IMO will make API look a bit weird:

HTTP.get("https://a.example.com/foo/bar", :host => "b.example.com")

So in the example above it will mean connect to a.example.com, but set SNI to b.example.com and send HTTP request:

GET /foo/bar
Host: b.example.com

[ixti]

@tarcieri isn't ssl.hostname = ... is what used for both SNI and certificate verification?

https://github.com/httprb/http/blob/4bc1223c5bbaf85dfc142f353a30594242c7a4bb/lib/http/timeout/null.rb#L32-L34

Probably because of being pretty weak in this topic I thought that it's used for both. :D

[tarcieri]

@tarcieri isn't ssl.hostname = ... is what used for both SNI and certificate verification?

NOPE! ssl.hostname = (at least I checked) ONLY sets SNI, and has no effect on hostname verification.

[ixti]

Oh. At least I have correctly man'ed on SNI then :D In this case :host option seems more like the best option to me now... But I guess host verification should be done in any case, no?

[rchady]

@tarcieri I was thinking about that, but that IMO will make API look a bit weird:

HTTP.get("https://a.example.com/foo/bar", :host => "b.example.com")

So in the example above it will mean connect to a.example.com, but set SNI to b.example.com and send HTTP request:

GET /foo/bar
Host: b.example.com

In my humble opinion, this is very clear as to what it is doing and would solve my use case perfectly. It also would work for the use case where you want/need to connect to an IP address, but connect to a given vhost over SSL. I've seen multiple requests from people for that use case, but usually to no avail.

[ixti]

use case where you want/need to connect to an IP address, but connect to a given vhost over SSL

I agree about that. Just thinking that:

HTTP.connect("1.2.3.4").get("https://abc.example.com/foobar")

# or

HTTP.get("https://abc.example.com/foobar", :connect => "1.2.3.4")

looks easier to understand than:

HTTP.get("https://1.2.3.4/foobar", :host => "abc.example.com")

[ixti]

Now as I wrote this and looking at the pseudo code, both variants seems OK to me.

[tarcieri]

CONNECT already has a special meaning in HTTP:

https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT

[ixti]

@tarcieri Yeah I know. Didn't found any good name - so decided that for sake of example it gives an overall idea. :D

[rchady]

As a side note I wanted to say, I just got done porting some other code of mine from net/http to httprb... OMG what a difference! Thank you so much for this... keep it up.

[cben]

HTTP.get("https://abc.example.com/foobar", :connect => "1.2.3.4")

looks easier to understand than:

HTTP.get("https://1.2.3.4/foobar", :host => "abc.example.com")

I want to second this :100: Semantically, when I have a URL like https://abc.example.com/foobar, that implies I want to ask the server (via SNI and Host header) for abc.example.com, and I want to verify certificate for abc.example.com. The only deviation here is accessing a different IP address, and that should be the out-of-URL parameter. It can be thought of as not interfering in HTTP[S] semantics, only in DNS resolution (similar to say /etc/hosts). FWIW, curl views it similarly:

curl https://abc.example.com/foobar --resolve abc.example.com:443:1.2.3.4
curl https://abc.example.com/foobar --connect-to abc.example.com:443:1.2.3.4:443

The :host option is weirder, as I supply URL of https://1.2.3.4/foobar but am requesting (and securing) a different URL with https://abc.example.com/foobar. While one could mechanically view it as "I'm fetching https://1.2.3.4/foobar, just customizing (1) Host header, (2) SNI (3) cert checking", this is a low-level view predating virtual hosts & SNI...

• I'm ignoring here the fact it's already possible to pass custom Host header (as any other header).

---

Naming ideas:

HTTP.get("https://abc.example.com/foobar", :address => "1.2.3.4")
HTTP.get("https://abc.example.com/foobar", :IP => "1.2.3.4")

vs

HTTP.get("https://1.2.3.4/foobar", :origin => "abc.example.com")

(using the browser security term "origin" roughly meaning "domain + strong guarantee it really came from that domain"...)

[tarcieri]

HTTP.get("https://abc.example.com/foobar", :connect => "1.2.3.4")

looks easier to understand than:

HTTP.get("https://1.2.3.4/foobar", :host => "abc.example.com")

I want to second this 💯

CONNECT is an HTTP method for fetching a resource through a proxy.

If I saw anything resembling:

HTTP.get("https://abc.example.com/foobar", :connect => "1.2.3.4")

...in any other HTTP library, my assumption would be 1.2.3.4 is an HTTP CONNECT proxy through which https://abc.example.com/foobar is being fetched.

I would personally be extremely confused if the behavior was anything but the above. Given the special meaning of CONNECT in the protocol, if we're going to use that word anywhere, my extremely strong preference would be to use it for that purpose, and that purpose alone, to the exclusion of all others.

Circling back, the name of this issue is:

Control SNI hostname used?

Ultimately it's the SNI hostname that people are desiring to override, and therefore I think that's the part that needs to be parameterized.

Neither :address or :IP make sense to me. Address of what? :IP in particular is bad because people may still wish to connect through an alternative DNS name.

[ixti]

After revisiting this thread - I changed my mind (partially) and think that :host => ... option looks better indeed, but I don't see it fitting future session object though. It seems to me that there should be some sort of resolver DI for this thing, or host-rewrite mapping some sort of... o_O

[colemannugent]

So I ran into a similar issue where I needed to talk with an endpoint without setting the SNI extension. For anyone else in that boat I offer this monkey patch:

module OpenSSL
  module SSL
    class SSLSocket
      undef_method :hostname=
    end
  end
end

With that said, I think it should be pretty easy to address both my admittedly non-standard use-case and @rchady 's more sensible use-case with one change. Borrowing from @ixti 's original idea, we add an additional option to HTTP::Options that controls the SNI extension. This could work something like this:

# Different SNI than host
HTTP.get("https://a.example.com/foo/bar", sni: "b.example.com")

# Disable SNI
HTTP.get("https://a.example.com/foo/bar", sni: nil)

As far as I can tell, this would require a change to start_tls in HTTP::Connection to pass the SNI option to start_tls in HTTP::Timeout::Null. Adding a sni keyword argument should allow this change to not affect any other code. So start_tls would look something like this:

# Configures the SSL connection and starts the connection
def start_tls(host, ssl_socket_class, ssl_context, sni: host)
...
  @socket.hostname = sni if sni && @socket.respond_to?(:hostname=)
...

@tarcieri, would you consider merging something like this if I put together a PR an updated the docs?

[tarcieri]

Sure, that seems fine to me so long as it's otherwise non-breaking (i.e. SNI gets set automatically by the hostname otherwise).