add --resolve-host flag to force DNS resolution to explode multiple A/AAAA records

[fatpat]

when s3.xxxx.org has multiple A/AAAA records, I would want warp to explode it to all the IPs without having to specifying all of them by hand.

[fatpat]

Requires formatting and a small help text tweak.

done and I also hide the --resolve-host flag as it is quite specific

[fatpat]

what is the purpose of this flag?

lets assume that I want to target https://s3.xxxx.org and s3.xxxx.org resolves to multiple A/AAAA records (a.b.c.1, a.b.c.2 and a.b.c.3). When setting --host s3.xxxx.org the resolution of the domain is done by the s3 client and it implies cache. Which means that the resolution will be done once for all the concurrent requests and they will all target the same A/AAAA record. We would maybe expect warp to target each IPs from the records instead of just one.

with the resolve-host flag, the resolution is done when parsing the --host flag and it's converted to a list. It's like if warp was called with -host a.b.c.1,a.b.c.2,a.b.c.3.

When using TLS, changing a DNS to an IP could break certificate verification. In this case the --insecure flag must be used as a workaround.

[harshavardhana]

what is the purpose of this flag?

lets assume that I want to target https://s3.xxxx.org and s3.xxxx.org resolves to multiple A/AAAA records (a.b.c.1, a.b.c.2 and a.b.c.3). When setting --host s3.xxxx.org the resolution of the domain is done by the s3 client and it implies cache. Which means that the resolution will be done once for all the concurrent requests and they will all target the same A/AAAA record. We would maybe expect warp to target each IPs from the records instead of just one.

with the resolve-host flag, the resolution is done when parsing the --host flag and it's converted to a list. It's like if warp was called with -host a.b.c.1,a.b.c.2,a.b.c.3.

When using TLS, changing a DNS to an IP could break certificate verification. In this case the --insecure flag must be used as a workaround.

In Go net.Dial automatically dials the fastest connection, have you explored writing this with a custom DialContext instead? @fatpat

var rng = rand.New(rand.NewSource(time.Now().UTC().UnixNano()))

type dialContext func(ctx context.Context, network, address string) (net.Conn, error)

func newCustomDialContext(dialTimeout, dialKeepAlive time.Duration) dialContext {
        return func(ctx context.Context, network, addr string) (net.Conn, error) {
                dialer := &net.Dialer{
                        Timeout:   dialTimeout,
                        KeepAlive: dialKeepAlive,
                }

                host, port, err := net.SplitHostPort(addr)
                if err != nil {
                        return nil, err
                }

                addrs, err := net.LookupHost(host)
                if err != nil {
                        addrs = []string{host}
                }

                for i := range addrs {
                        addrs[i] = net.JoinHostPort(addrs[i], port)
                }

                return dialer.DialContext(ctx, network, addrs[rng.Intn(len(addrs))])
        }
}

[fatpat]

what is the purpose of this flag?

lets assume that I want to target https://s3.xxxx.org and s3.xxxx.org resolves to multiple A/AAAA records (a.b.c.1, a.b.c.2 and a.b.c.3). When setting --host s3.xxxx.org the resolution of the domain is done by the s3 client and it implies cache. Which means that the resolution will be done once for all the concurrent requests and they will all target the same A/AAAA record. We would maybe expect warp to target each IPs from the records instead of just one. with the resolve-host flag, the resolution is done when parsing the --host flag and it's converted to a list. It's like if warp was called with -host a.b.c.1,a.b.c.2,a.b.c.3. When using TLS, changing a DNS to an IP could break certificate verification. In this case the --insecure flag must be used as a workaround.

In Go net.Dial automatically dials the fastest connection, have you explored writing this with a custom DialContext instead? @fatpat

var rng = rand.New(rand.NewSource(time.Now().UTC().UnixNano()))

type dialContext func(ctx context.Context, network, address string) (net.Conn, error)

func newCustomDialContext(dialTimeout, dialKeepAlive time.Duration) dialContext {
        return func(ctx context.Context, network, addr string) (net.Conn, error) {
                dialer := &net.Dialer{
                        Timeout:   dialTimeout,
                        KeepAlive: dialKeepAlive,
                }

                host, port, err := net.SplitHostPort(addr)
                if err != nil {
                        return nil, err
                }

                addrs, err := net.LookupHost(host)
                if err != nil {
                        addrs = []string{host}
                }

                for i := range addrs {
                        addrs[i] = net.JoinHostPort(addrs[i], port)
                }

                return dialer.DialContext(ctx, network, addrs[rng.Intn(len(addrs))])
        }
}

@harshavardhana : nope. But I'll definitely look into it. It could be a cleaner solution.

[fatpat]

@harshavardhana your code works like a charm. This is much cleaner and it works with --tls without the need to add --insecure :+1:

[klauspost]

@harshavardhana @fatpat With "exploded" hosts, we will get stats for each host and we control the hosts selection, so I think the PR as proposed gives more value than a custom dialer.

[harshavardhana]

@harshavardhana @fatpat With "exploded" hosts, we will get stats for each host and we control the hosts selection, so I think the PR as proposed gives more value than a custom dialer.

yes I realized @klauspost since it randomizes it may hit the same server and the weighted distribution would have an effect.

Sorry @fatpat we may have to go back.

We can think about adding weighted distribution directly to Dialer itself if needed but that may be a bit more time consuming implementation and might not be needed at this point in time.

[fatpat]

ok, patch has been rolled back.

I added the --shuffle-hosts flag to control whether or not we want to shuffle the host list.

[fatpat]

@klauspost can you review and maybe merge this please ?