grpc-go icon indicating copy to clipboard operation
grpc-go copied to clipboard

Grpc DNS resolver timing out for simple localhost lookup

Open peterchenadded opened this issue 6 months ago • 4 comments

What version of gRPC are you using?

1.71.1

What version of Go are you using (go version)?

1.24.2 Darwin/arm64

What operating system (Linux, Windows, …) and version?

Mac m1 Sequoia 15.5

What did you do?

This is a follow up of https://github.com/grpc/grpc-go/issues/7429#issuecomment-2905313867

Essentially tried running:

https://github.com/grpc/grpc-go/blob/master/examples/helloworld/greeter_client/main.go

But got:

rpc error: code = DeadlineExceeded desc= context deadline exceeded

This was with

localhost:4317

It worked fine with

127.0.01:4317

What did you expect to see?

I expected both localhost and 127.0.01 to work.

What did you see instead?

Only 127.0.0.1 worked

I had to remove the two DNS nameservers entries on my machine to get localhost to work. But this is not an appropriate fix because I needed root to do this and breaks other functionality that does need it.

I could have also
used passthrough://localhost:4317, but for some libraries such as OTEL exporters that use grpc it is currently not possible to pass through the scheme.

I tried net.DefaultResolver.LookupIP(ctx, 'ip', 'localhost') and it resolved with no issues quickly.

The issue seems to be that grpc DNS resolver is trying to contact the name servers even for simple /etc/hosts entries that net.DefaultResolver has no issues working with.

peterchenadded avatar May 24 '25 00:05 peterchenadded

Ok after a deep dive into the code I can see

  1. The code that does the resolution is at

    https://github.com/grpc/grpc-go/blob/master/internal/resolver/dns/dns_resolver.go#L336

  2. The lookupHost is working since it is using net.DefaultResolver

  3. but after that it tries to do

    lookupTXT on _grpc_config.localhost

    That is timing out after 5 seconds with

    I/o timeout

  4. So question is how to disable that Txt lookup

    grpc.WithDisableServiceConfig()

Is it possible to have an environment variable to turn this off, to work with problematic nameservers?

peterchenadded avatar May 24 '25 01:05 peterchenadded

Hi @peterchenadded, there isn't a way to selectively disable service config lookups for hosts. You will need to use the WithDisableServiceConfig() dial option or provide an IP address as the target (e.g: "127.0.0.1" instead of "localhost") to bypass service config lookups. To allow users to configure gRPC channels, libraries usually accept their own options structs which they convert to gRPC DialOptions internally. GCP client libraries are an example of this pattern.

arjan-bal avatar May 26 '25 06:05 arjan-bal

Thanks @arjan-bal problem is I don't have direct access to change the client options for the library am using e.g. otel grpc.

If there no plans to add a environment variable to disable it, we can close this issue off.

I will have to live with 127.0.0.1 for my local Mac setup.

peterchenadded avatar May 26 '25 07:05 peterchenadded

It seems like we should probably be doing something here to help. Reopening as we discuss internally what our options are.

dfawley avatar Jun 02 '25 20:06 dfawley

Thanks @dfawley, do we know what version this will be released in?

peterchenadded avatar Jul 01 '25 22:07 peterchenadded

@peterchenadded we already cut the release branch for 1.74, which is happening next week. So this would make it into 1.75 which would release on/around Aug 19th.

dfawley avatar Jul 01 '25 23:07 dfawley