High connect latency in Windows

[rodem]

Simple example code:

func main() {
	go func() {
		http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
			json.NewEncoder(w).Encode("Hello, world!\n\n")
		})
		http.ListenAndServe(":7000", nil)
	}()

	fasthttp.ListenAndServe(":8080", func(ctx *fasthttp.RequestCtx) {
		fmt.Fprintf(ctx, "Hello, world!\n\n")
	})
}

Curl options:

time_namelookup:  %{time_namelookup}\n
     time_connect:  %{time_connect}\n
  time_appconnect:  %{time_appconnect}\n
 time_pretransfer:  %{time_pretransfer}\n
    time_redirect:  %{time_redirect}\n
time_starttransfer:  %{time_starttransfer}\n
                  ----------\n
       time_total:  %{time_total}\n

Curl for net/http:

>curl -w "@curl-format.txt" -o NUL -s http://localhost:7000/
    time_namelookup:  0,016000
       time_connect:  0,016000
    time_appconnect:  0,000000
   time_pretransfer:  0,031000
      time_redirect:  0,000000
 time_starttransfer:  0,031000
                    ----------
         time_total:  0,031000

Curl for fasthttp:

>curl -w "@curl-format.txt" -o NUL -s http://localhost:8080/
    time_namelookup:  0,000001
       time_connect:  0,203000
    time_appconnect:  0,000000
   time_pretransfer:  0,203000
      time_redirect:  0,000000
 time_starttransfer:  0,203000
                    ----------
         time_total:  0,203000

Same results via php 7 file_get_contents. go version go1.13.8 windows/amd64 windows 10/ windows server 2012R2

[erikdubbelboer]

I'm afraid I don't have a windows machine to try and test this on. @kirillDanshin do you?

[avkhra]

btw on my windows the same, on linux better , but anyway net/http faster

[kirillDanshin]

@avkhra do you have the same results if you put net/http and fasthttp servers in different binaries?

[Fenny]

I tested this with two different binaries on Windows 10 running Go v1.14.1

// net/http
time_namelookup:  0.015000
     time_connect:  0.015000
  time_appconnect:  0.000000
 time_pretransfer:  0.031000
    time_redirect:  0.000000
time_starttransfer:  0.031000
                  ----------
       time_total:  0.031000

// fasthttp
time_namelookup:  0.015000
     time_connect:  0.234000
  time_appconnect:  0.000000
 time_pretransfer:  0.234000
    time_redirect:  0.000000
time_starttransfer:  0.234000
                  ----------
       time_total:  0.234000

[niallnsec]

I tried with the latest version of fasthttp and Go (1.16.3) and this is still reproducible.

I believe the issue is with the Windows networking stack, in particular the way in which windows handles IPv4/6 dual stacks and as such is not necessarily an issue for fasthttp to address.

I found 2 ways to remove the additional latency observed during connect on Windows. I was investigating the differences between the two packages and the only notable difference I could see in the accept portions of the code respectively was that net/http was listening using "tcp" as the network while fasthttp was using "tcp4". I changed line 1493 in server.go to listen on "tcp" rather than "tcp4" in order to test if this was causing the difference and re ran the curl commands.

After making the change, the connect time differences observed between net/http and fasthttp was completely resolved. This led me to investigate the role that dual stack IPv4/6 was playing in the issue, as IPv6 on windows has been known to cause performance issues in other places. Notably there has long been problems with increased DNS connect latency when IPv6 is enabled on Windows.

I reverted the change and tried instead changing the curl invocation to use the IPv4 loopback interface explicitly: curl -w "@curl-format.txt" -o NUL -s http://127.0.0.1:8080/ The result was the following output:

  time_namelookup:  0.000001
     time_connect:  0.000001
  time_appconnect:  0.000000
 time_pretransfer:  0.000001
    time_redirect:  0.000000
time_starttransfer:  0.000001
                  ----------
       time_total:  0.000000

and for net/http:

  time_namelookup:  0.000001
     time_connect:  0.000001
  time_appconnect:  0.000000
 time_pretransfer:  0.000001
    time_redirect:  0.000000
time_starttransfer:  0.016000
                  ----------
       time_total:  0.016000

So overall I think this confirms that the issue is not in fasthttp or Go, but rather in the way Windows is handling the dual stack IPv4/6 connect attempts. I suspect that the resolution of localhost is returning ::1 as the first response causing the system to attempt to connect to the IPv6 loopback address first, when this fails Windows is then transparently attempting to connect to the IPv4 address, wasting ~200ms in the process.

As for the resolution to this issue, perhaps it is worth a note in the documentation about this behaviour on Windows. I did also wonder if it is strictly necessary for the default ListenAndServe method to listen on IPv4 only? Is there an advantage to disabling IPv6 which I am not aware of? If not then changing the default listener to use "tcp" as opposed to "tcp4" would address this issue and improve the functioning of fasthttp on Windows, as well as allowing the default fasthttp to server to listen on IPv6 addresses with no real downside (that I am aware of).

[erikdubbelboer]

Great research @niallnsec. I don't think we can just changed it to tcp from tcp4. This is a big change that might break people's code. I think that this issue and everything described in it is probably already enough documentation about this problem. People will find it when they search. Since the problem is window trying IPV6 for 200ms and then switching to IPV4 I don't think adding documentation about this to the Listen functions make sense. They already properly document that it will only listen on IPV4 and that you need make your own listener for IPV6.

[Murtagy]

What is the default way to handle this in Windows machine? Is that patching lib code?

I am not very experienced in the subject and it took me a while to see that there is nothing in my handler which makes the server slow and why initial net/http version is so much faster. I would say that a line in docs can save some time to humanity.

[erikdubbelboer]

You don't need to patch the lib code, just create your own listener using tcp and pass it to Server.Serve