Importer: Connection to API failed

[benjamin-lang]

Describe the bug When configuring an importer (reflection) with our grpc endpoint, the connection to the api fails.

To Reproduce Steps to reproduce the behavior:

1. Configuring importer
2. Use Endpoint with TLS enabled
3. Use Root Certificate for the endpoint
4. Hit 'Save' button

Expected behavior Connection sucessful established

Screenshots Grpc.Core.RpcException: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: Requesting HTTP version 2.0 with version policy RequestVersionOrHigher while unable to establish HTTP/2 connection.", DebugException="System.Net.Http.HttpRequestException: Requesting HTTP version 2.0 with version policy RequestVersionOrHigher while unable to establish HTTP/2 connection. at System.Net.Http.HttpConnectionPool.ThrowGetVersionException(HttpRequestMessage request, Int32 desiredVersion) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Grpc.Net.Client.Internal.GrpcCall 2.RunCall(HttpRequestMessage request, Nullable 1 timeout)") at Grpc.Net.Client.Internal.HttpContentClientStreamWriter 2.WriteAsyncCore[TState](Func 5 writeFunc, TState state) at Kreya.Grpc.Core.Importer.ServerReflection.GrpcServerReflectionImporter.ReadAllServices(AsyncDuplexStreamingCall 2 call) at Kreya.Grpc.Core.Importer.ServerReflection.GrpcServerReflectionImporter.ImportViaServerReflection(GrpcServerReflectionImporterOptions options)

Environment (if possible, copy the information from the error dialog or the About menu):

• OS: Windows 10
• Kreya App Version: Created on 11.07.2022

Additional context

• Connection via grpcurl commandline tool works as expected
• Api is provided via openshift and accessed via openshift route with valid tls certificates and reencrypt route configuration

Thanks in advance

• Ben

[CommonGuy]

I suspect proxy issues (probably an enterprise HTTP proxy that does not support HTTP/2 correctly, which is required for gRPC to work). grpcurl does not support proxies, so that's probably why it works.

Could you try adding the no_proxy environment variable and then restarting Kreya if it's open? For example no_proxy=example.com:5000. This should cause Kreya to ignore the proxy for the specified host.

[benjamin-lang]

Hi that would be a possible reason, but i cannot get it to run. I tried to set the NO_PROXY variable in windows system env and via following batch file:

set NO_PROXY=our.dns.com
start "" "c:\program files\kreya\kreya.exe"

But still the same result ... any other ideas or debugging options ?

[CommonGuy]

Your server reflection URL starts with https://, right?

Could you also try something along this way?

set HTTP_PROXY=
set HTTPS_PROXY=
set ALL_PROXY=
start "" "c:\program files\kreya\kreya.exe"

[benjamin-lang]

Your server reflection URL starts with https://, right?

Yes something like https://openshift-route.app.our.internal-domain.com

That variant with everything set empty, did not solve the problem or change the error message.

[CommonGuy]

I could only find two ways to reproduce this:

• By setting the environment variable DOTNET_SYSTEM_NET_HTTP_SOCKETSHTTPHANDLER_HTTP2SUPPORT=false
• By setting HTTP_PROXY and HTTPS_PROXY to http://example.com (or any HTTP URL). Using https://example.com works/shows another error message.

Does one of these apply to you?

[benjamin-lang]

Does one of these apply to you?

No not really. Maybe it has something todo with our openshift setup: https://docs.openshift.com/container-platform/4.8/networking/ingress-operator.html

In chapter: Enabling HTTP/2 Ingress connectivity there is a important note:

For non-passthrough routes, the Ingress Controller negotiates its connection to the application independently of the connection from the client. This means a client may connect to the Ingress Controller and negotiate HTTP/1.1, and the Ingress Controller may then connect to the application, negotiate HTTP/2, and forward the request from the client HTTP/1.1 connection using the HTTP/2 connection to the application. This poses a problem if the client subsequently tries to upgrade its connection from HTTP/1.1 to the WebSocket protocol, because the Ingress Controller cannot forward WebSocket to HTTP/2 and cannot upgrade its HTTP/2 connection to WebSocket. Consequently, if you have an application that is intended to accept WebSocket connections, it must not allow negotiating the HTTP/2 protocol or else clients will fail to upgrade to the WebSocket protocol.

Could this maybe the root of the problem ?

[CommonGuy]

I'm really at a loss here. It may be a misconfigured HAProxy issue, but then grpcurl shouldn't work either.

Could you try the Kreya importer without a certificate? Do you get a different error?

If nothing works, could you post a screenshot of your importer configuration (with the domain blacked out)? And also a grpcurl your-domain.com list command (with whatever options you need to provide to get it to work) that works on your machine? grpcurl domain.com list uses server reflection and should roughly behave the same way as Kreya.

[benjamin-lang]

Could you try the Kreya importer without a certificate? Do you get a different error? No, same error.

Here the importer configuration:

And here the grcpcurl call in git bash:

I noticed that grpcurl with "https://" scheme does not work with: dial tcp: lookup tcp///our.domain.de: getaddrinfow: the specified class was not found.

[CommonGuy]

Do you have access to curl on your machine? Could you show the full output (domain name can be removed) from this command?

curl -sIv https://your-domain.com

Please also test whether https://example.com uses HTTP/2, as curl on some Windows machines does not yet support HTTP/2.

[benjamin-lang]

Hi, sorry for late response:

$ curl -sIv https://*******.ocp.our-domain.com/
* Uses proxy env variable no_proxy == 'ocp.our-domain.com'
*   Trying 10.64.1.15:443...
* Connected to *******.ocp.our-domain.com (10.64.1.15) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: C:/***/portable/PortableGit/mingw64/ssl/certs/ca-bundle.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self signed certificate in certificate chain
* Closing connection 0

The used curl version was:

$ curl --version
curl 7.81.0 (x86_64-w64-mingw32) libcurl/7.81.0 OpenSSL/1.1.1m (Schannel) zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 libidn2/2.3.1 libssh2/1.10.0 nghttp2/1.46.0
Release-Date: 2022-01-05
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL SSPI TLS-SRP zstd

[CommonGuy]

It looks like the endpoint supports HTTP/2 correctly (as indicated by ALPN, offering h2).

Would you mind trying another theory? It may be that Kreya (or .NET rather) uses the system proxy configuration rather than the configuration from the environment variables. To test that, could you set both HTTPS_PROXY=your-proxy.domain and NO_PROXY=*******.ocp.our-domain.com? Because if neither HTTP_PROXY, HTTPS_PROXY or ALL_PROXY have been set, it uses the system proxy configuration.

[benjamin-lang]

I tried to start it with the following batch file:

set HTTP_PROXY=
set HTTPS_PROXY=http://proxy.our-domain.com:8080
set NO_PROXY=ocp.our-domain.com

set

start "" "c:\program files\kreya\kreya.exe" 

pause

But the Result is still the same. It does not seem to take, whatever we set.

[CommonGuy]

Could you try https://grpcb.in:9001 as the importer URL (without any certificate) to see whether other gRPC services work?

In addition, could you try a GET https://your-endpoint.domain/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo REST (not gRPC) request with Kreya? You may need to configure your certificate in the "Auth" tab.

[benjamin-lang]

Hi i tried https://grpcb.in:9001, but it did not succeed. I suppose the reason is that our proxy does not support http2. The endpoint https://your-endpoint.domain/grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo via REST does not exist.

What I could verify, is that connection with Kreya with a local service without tls does work with reflection.

[CommonGuy]

Yes, it looks like your company network/proxy is the root cause of the failure. I don't think we can do anything on our side to solve this issue

[benjamin-lang]

Yes, it looks like your company network/proxy is the root cause of the failure. I don't think we can do anything on our side to solve this issue

If that would be the case, I don't think grpcurl and postman would run without problems. But at the moment i have no further ideas either.

[benjamin-lang]

I still don't get it to work, if you (someday) implement a developer flag (verbose logging), tell me. Maybe we could log the and solve the problem then.

PS: I found someone with the same problem in the dot.net issues: https://github.com/dotnet/runtime/issues/56217

[CommonGuy]

Could you check whether you have any special registry settings in HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL? In https://github.com/dotnet/runtime/issues/62473#issuecomment-988268102, a special Schannel configuration was the cause for this problem.

The Schannel configuration is used by .NET, but not by curl/grpcurl, which could explain the different behaviour.

[benjamin-lang]

I looked into it, but there are sadly no special keys in place in the schannel configuration

[CommonGuy]

Could you try the following in the importer configuration?

1. Change "Server certificate validation" to "Disabled"
2. Remove the "Certificate"
3. Hit "Save"

If this does not work, try to change the "Endpoint" from https:// to http:// (with no certificate specified).

[benjamin-lang]

Hi @CommonGuy,

i tried again with the current version, but the same behaviour wether server certificate validation is enabled or disabled. If I change the url to http I get a timeout, there is no route configured with http access in our cluster.

[CommonGuy]

Thanks for trying!

The most likely cause is that the server does not support HTTP/2 (or rather ALPN) correctly. Kreya tries to establish a HTTP/2 connection (required for gRPC) with the server, which requires a successful application-layer protocol negotiation (ALPN). However, this seems to fail.

The reason why other clients such as grpcurl work is because they do not conform to the HTTP/2 standard (or rather, their language/framework is not conformant). They use HTTP/2 with "prior knowledge", disregarding the ALPN result. According to the standard, this approach should only be used for http:// URLs. See https://www.rfc-editor.org/rfc/rfc7540#section-3.3. The standard library of C# (the language Kreya is written in) is more strict and needs a successful ALPN before it can use a HTTP/2 connection.

I suspect you need to configure something in your ingress, so that the ALPN works. If you are able to run C#, here is a small example that reproduces the problem:

using System;
using System.Net.Http;        
using System.Threading.Tasks;

public class Program
{
    public static async Task Main()
    {
        var client = new HttpClient();
        client.DefaultRequestVersion = new Version(2, 0);
        client.DefaultVersionPolicy = HttpVersionPolicy.RequestVersionOrHigher;
        await client.GetStringAsync("https://your-url.com");
        Console.WriteLine("HTTP/2 connection works");
    }
}

If it crashes, HTTP/2 does not work.

[benjamin-lang]

I found the following entry for openshift in combination with ALPN: https://cloud.redhat.com/blog/grpc-or-http/2-ingress-connectivity-in-openshift

Known Limitation For re-encrypt routes, the Ingress Controller negotiates its connection to the application independently of the connection from the client. This means a client may connect to the HAProxy Ingress Controller and negotiate HTTP/1.1, and the Ingress Controller may then connect to the application service, negotiate HTTP/2, and forward the request from the client HTTP/1.1 connection using the HTTP/2 connection to the application. This poses a problem if the client subsequently tries to upgrade its connection from HTTP/1.1 to the WebSocket protocol, because the Ingress Controller cannot forward WebSocket to HTTP/2 and cannot upgrade its HTTP/2 connection to WebSocket. Consequently, if you have an application that is intended to accept WebSocket connections, it must not allow negotiating the HTTP/2 protocol or else clients will fail to upgrade to the WebSocket protocol.

There is already a reported issue for this limitation in HAProxy GitHub repo. You can further track this issue if it’s really a blocker for you and you want to check the progress for it’s fix: GitHub Issue

Maybe that could be the reason ... I will try to find out which version of HAProxy is used in our cluster.

[benjamin-lang]

Hi, we tried Kreya after one year again, some infrastructure changed inside our company and I hoped maybe it would run now. But I still cannot get it to run. Is there any log/verbose option in kreya inbetween which I could activate ?

[CommonGuy]

If you cannot get the gRPC server reflection to work with https://grpcb.in:9001, it must be something on your computer/network that interfers, since that endpoint works for everyone else. We do not have more detailed logging... I suspect a proxy, antivirus or something else to intercept the traffic and incorrectly handle HTTP/2 connections.

[benjamin-lang]

Sure in every big company is a proxy to www to prevent security issues. I will reach out for our network department and I will try to get the proxy opened.

[CommonGuy]

Did you manage to resolve this issue?

[benjamin-lang]

No sorry I tried it multiple times, but we did not manage to resolve the problem. We end up using postman these days, sorry. From my perspective you can close this issue. Thanks

[CommonGuy]

Thanks for your patience and detailed explanations. We are sorry that Kreya does not work for you