Grpc.Net >= 2.44.0 does not pick up HTTPS_PROXY environment variable

[odin568]

What version of gRPC and what language are you using?

Issue detected initally with 2.46.0 but started with 2.44.0. Behavior was fine until 2.43.0 (tested from 2.38.0 on)

What operating system (Linux, Windows,...) and version?

Linux container - original Microsoft ASP.NET runtime container (latest)

What runtime / compiler are you using (e.g. .NET Core SDK version dotnet --info)

 Version:   6.0.302
 Commit:    c857713418
Runtime Environment:
 OS Name:     ubuntu
 OS Version:  22.04
 OS Platform: Linux
 RID:         ubuntu.22.04-x64
 Base Path:   /usr/share/dotnet/sdk/6.0.302/
global.json file:
  Not found
Host:
  Version:      6.0.7
  Architecture: x64
  Commit:       0ec02c8c96
.NET SDKs installed:
  6.0.302 [/usr/share/dotnet/sdk]
.NET runtimes installed:
  Microsoft.AspNetCore.App 6.0.7 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 6.0.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

What did you do?

Issue came up when I upgraded Google.Cloud.PubSub.V1 from 2.10.0 to 3.0.0. This came with a change to replace the default Grpc.Core implementation with Grpc.Net. Thereby I first got connection issues as the grpc_proxy environment variable is of course not used anymore. I then changed it to HTTPS_PROXY (and HTTP_PROXY) and expected it to work but anyhow it was just stuck/hanging until the task was forcably canceled.

I then found a related issue in the google-cloud-dotnet repository here: https://github.com/googleapis/google-cloud-dotnet/issues/8834

The discussions here are quite lengthy as we tried out many many variants until we came to the point to see that the issue seems to be in Grpc.Net and that the environment variable was properly taken into account until 2.43.0 and ignored with 2.44.0 on. Interestingly, folks at the google repo were not able to reproduce it, it worked on their side but there is somebody else reporting the same problem. So it seems to be a special case in some configuration - where we don't know the original reason for it.

Our testcode was mainly this at the end:

static void TestGrpcNetClient_SocketHttpHandler() =>
    TestGrpcNetClient(
        GrpcChannel.ForAddress("https://pubsub.googleapis.com",
            new GrpcChannelOptions { HttpHandler = new SocketsHttpHandler() }),
        "GrpcNetClient_SocketHttpHandler");

static void TestGrpcNetClient(GrpcChannel channel, string testName)
{
    var callInvoker = channel.CreateCallInvoker();
    var marshaller = new Marshaller<string>(Encoding.UTF8.GetBytes, Encoding.UTF8.GetString);
    var method = new Method<string, string>(MethodType.Unary,
        "test-service", "test-method",
        marshaller, marshaller);
    try
    {
        Console.WriteLine($"Starting request for {testName}.");
        var response = callInvoker.BlockingUnaryCall(method, null,
            default, "test-request");
        Console.WriteLine($"Got response {response} for {testName}.");
    }
    catch (Exception e)
    {
        Console.WriteLine($"{testName} failed.");
        Console.WriteLine(e);
    }
}

What did you expect to see?

With the above code, I would like to see that the code returns with the failure that this method/service of course does not exist instantly more or less.

What did you see instead?

Instead it "hangs" with 2.44.0 onwards as it does not take the proxy into consideration specified by environment variable.

For details see the linked ticket https://github.com/googleapis/google-cloud-dotnet/issues/8834 I am also happy to continue support here, just need to tell that I am off for vacation later on until 22nd of August. I hope the so far specified information help already, otherwise, please don't close this issue and I will provide more information that you need after that time. Thank you very much!

Anything else we should know about your project / environment?

I run application in k8s environment in corporate environment which requires proxy to reach internet.

[rafikiassumani-msft]

@odin568 Try replacing SocketsHttpHandler with HttpClientHandler in the following code:

 TestGrpcNetClient(
        GrpcChannel.ForAddress("https://pubsub.googleapis.com",
            new GrpcChannelOptions { HttpHandler = new SocketsHttpHandler() }),
        "GrpcNetClient_SocketHttpHandler");

Using HttpClientHandler disables some custom logic the channel has for creating the connection.

We'll look into whether there is a way to make this work with SocketsHttpHandler.

[amanda-tarafa]

@jtattermusch as FYI

@odin568 I'm taking the liberty to reply here as I know you are out on vacation.

@rafikiassumani-msft We tried that in https://github.com/googleapis/google-cloud-dotnet/issues/8834#issuecomment-1197968912 and it does work.

The problem is that @odin568 is not using Grpc.Net.Client directly, but throught the Google .NET client libraries which are the ones relying on Grpc.Net.Client. In Google .NET client libraries we offer a way to configure the underlying GrpcChannel, but now we would be asking users to bypass Google .NET client libraries defaults so they can bypass Grpc.Net.Client defaults, and that seems a bit too much for something that in the end is a regression from Grcp.Net.Client 2.43.0 to Grcp.Net.Client 2.44.0. And similar will happen for anyone building libraries on top of Grpc.Net.Client.

Maybe a quick fix, at least to get the default channel working, would be to use HttpClient.DefaultProxy in the code below to detect if if the default proxy applies to channel URI, and in that case avoid a SocketsHttpHandler and fall back to an HttpClientHandler instead?

https://github.com/grpc/grpc-dotnet/blob/a3717a358779178cfc4e03eeaaa8682ee433c0f1/src/Shared/HttpHandlerFactory.cs#L27-L38

Note that regardless, there would still be an issue with SocketsHttpHandler + Proxy, because even if the handler is configured explicitly with the proxy (i.e. not relying on the HTTP(S)_PROXY env variables) the proxy seems to be ignored completely on execution. See the code in the following comment, it fails (because it ignores the proxy) at least under certain circumstances.

https://github.com/googleapis/google-cloud-dotnet/issues/8834#issuecomment-1197997421

[jtattermusch]

@rafikiassumani-msft @JamesNK do we know what causing the SocketsHttpHandler in Grpc.Net.Client to ignore the HTTPS_PROXY setting? According to https://github.com/googleapis/google-cloud-dotnet/issues/8834#issuecomment-1197997421, HttpClient over SocketsHttpHandler does honor the HTTPS_PROXY, so there should be something we can do to be able to honor HTTPS_PROXY in Grpc.Net.Client as well.

[odin568]

Thanks all for investigation and fixes. Glad issue was found even though it is a special case hard to reproduce in different environmnents. What is the desired release plan/windows for such changes?