[UI] Relative Address for HealthCheckEndpoint with Kestrel at http://0.0.0.0:0

[Genmutant]

I'm trying to bind HealthCheckEndpoint using the relative address;

services.AddHealthChecksUI(setupSettings: settings => settings.AddHealthCheckEndpoint("ABC", "/health"));

but because it is configured to use http://0.0.0.0:0 in Kestrel, it throws an exception. I'm not hard set on using automatic port selection, though it is nice for testing. It throws the same exception when using a specific port, though.

2020-02-07 10:37:31.3948|WARN|Microsoft.AspNetCore.Server.Kestrel|Overriding address(es) 'https://localhost:5001'. Binding to endpoints defined in UseKestrel() instead.|
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: http://0.0.0.0:51713
2020-02-07 10:37:31.5178|INFO|Microsoft.Hosting.Lifetime|Now listening on: http://0.0.0.0:51713|
info: Microsoft.Hosting.Lifetime[0]
      Now listening on: https://0.0.0.0:51714
2020-02-07 10:37:31.5178|INFO|Microsoft.Hosting.Lifetime|Now listening on: https://0.0.0.0:51714|
fail: HealthChecks.UI.Core.HostedService.HealthCheckReportCollector[0]
      GetHealthReport threw an exception when trying to get report from /health configured with name TransactionServer.
System.Net.Http.HttpRequestException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')
 ---> System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')
   at System.Net.Dns.HostResolutionBeginHelper(String hostName, Boolean justReturnParsedIp, Boolean throwOnIIPAny, AsyncCallback requestCallback, Object state)
   at System.Net.Dns.BeginGetHostAddresses(String hostNameOrAddress, AsyncCallback requestCallback, Object state)
   at System.Net.Sockets.MultipleConnectAsync.StartConnectAsync(SocketAsyncEventArgs args, DnsEndPoint endPoint)
   at System.Net.Sockets.Socket.ConnectAsync(SocketType socketType, ProtocolType protocolType, SocketAsyncEventArgs e)
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HealthChecks.UI.Core.HostedService.HealthCheckReportCollector.GetHealthReport(HealthCheckConfiguration configuration)
2020-02-07 10:37:32.3037|ERROR|HealthChecks.UI.Core.HostedService.HealthCheckReportCollector|GetHealthReport threw an exception when trying to get report from /health configured with name TransactionServer.|System.Net.Http.HttpRequestException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')
 ---> System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')
   at System.Net.Dns.HostResolutionBeginHelper(String hostName, Boolean justReturnParsedIp, Boolean throwOnIIPAny, AsyncCallback requestCallback, Object state)
   at System.Net.Dns.BeginGetHostAddresses(String hostNameOrAddress, AsyncCallback requestCallback, Object state)
   at System.Net.Sockets.MultipleConnectAsync.StartConnectAsync(SocketAsyncEventArgs args, DnsEndPoint endPoint)
   at System.Net.Sockets.Socket.ConnectAsync(SocketType socketType, ProtocolType protocolType, SocketAsyncEventArgs e)
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HealthChecks.UI.Core.HostedService.HealthCheckReportCollector.GetHealthReport(HealthCheckConfiguration configuration)    at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at HealthChecks.UI.Core.HostedService.HealthCheckReportCollector.GetHealthReport(HealthCheckConfiguration configuration)

[tidusjar]

I'm having the same issue as this, did you find a solution?

[ildoc]

i've got the same error in appsettings.json

"HealthChecks-UI": {
    "HealthChecks": [
      {
        "Name": "MySelf",
        "Uri": "/hc"
      },
    [...]

[CarlosLanderas]

@Genmutant @tidusjar @ildoc , What operative system are you using, are you specifying kestrel to use 0.0.0.0?

When I configure relative uris my listening address is http://localhost:{port}

[ildoc]

I'm running my app from a linux docker container with all the kestrel default settings

[tidusjar]

I'm running on Windows, but this will apply to all OS's that net core supports. Kestrel is listening on http://*:5000

You can repro this using my repo if you want, just pull the sln down (branch = feature/v4) and un-comment the following lines: https://github.com/tidusjar/Ombi/blob/feature/v4/src/Ombi/Startup.cs#L84

[CarlosLanderas]

0.0.0.0 or * for IPv4 represents kestrel listening on all ips, and [::] is the IpV6 equivalent. However the IServerAddressesFeature just reports the plain text configuration and not the final ips, so we can say this feature can only work when mapping relative services to localhost

If the server addresses service receive something like: http://[::]:5000 it can't compose the final url.

I recommend you to setup the listerning urls in the docker container:

        public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
            WebHost.CreateDefaultBuilder(args)
            .UseUrls("http://localhost:5000")
            .UseStartup<Startup>();

in 3.X:

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseUrls("http://localhost:5000");
                    webBuilder.UseStartup<Startup>();
                });

[CarlosLanderas]

@rynowak any idea how can we use the IServerAddressFeature from AspNetcore to compose fully qualified urls from a relative path when kestrel is listening to all ips in ipv4/ ipv6?. Thanks!

[rynowak]

@rynowak any idea how can we use the IServerAddressFeature from AspNetcore to compose fully qualified urls from a relative path when kestrel is listening to all ips in ipv4/ ipv6?. Thanks!

There's no real way to do it without some guesswork. The problem is that the server doesn't know and doesn't care what hostname/ip the request arrives on 😆

Usually we avoid these problems in ASP.NET Core by using the hostname of the current request. That only helps if you have a request to use.

You could use something like Dns.GetHostname().

Or you could look at https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-3.1#host-filtering which is something we recommend people use in production deployments.

[CarlosLanderas]

Thank you very much @rynowak :). As this feature allows relative urls for the local UI, the GetHostName trick might work!.

[CarlosLanderas]

@rynowak this is working with 0.0.0.0:0 and *:{port}. Whenever the IServerAddressFeature reported host name is not resolved as a dns name we resolve using Dns.GetHostname. Looks good?

It resolves the dns hostname when hosts like [::], 0.0.0.0 are reported from the feature.

internal string AbsoluteUriFromRelative(string relativeUrl)
        {
            var targetAddress = AddressesFeature.Addresses.First();
            Uri.TryCreate(targetAddress, UriKind.Absolute, out var original);

            if (targetAddress.EndsWith("/"))
            {
                targetAddress = targetAddress[0..^1];
            }

            if (!relativeUrl.StartsWith("/"))
            {
                relativeUrl = $"/{relativeUrl}";
            }
            var hostCheck = Uri.CheckHostName(original.DnsSafeHost);

            if(hostCheck != UriHostNameType.Dns)
            {
                targetAddress = $"{original.Scheme}://{Dns.GetHostName()}:{original.Port}";
            }

            return $"{targetAddress}{relativeUrl}";
        }

[rynowak]

The code looks fine. The only concern that I'd have is that techniques like this aren't foolproof - there is no solution that is.

Consider what would happen if this were deployed to k8s. Someone navigates to http://health (based on Service). You (using Dns.GetHostName()) generate http://health-dashboard-dkfkdfk-39393 (based on Pod name). Now all your URLs are affinitized to the pod that generated them- if that pod is shut down then those URLs don't work.

Again, there's no foolproof method you can use here without making the user configure it. You have to know what DNS names are in use by all of your clients - this isn't information the server can guess.

[CarlosLanderas]

Would it be better just using DNS names in kestrel configuration?

I always configure apps to localhost and never found this problem before but based on the feedback in this issue it looks like there are several people listening on all ips.

[rynowak]

Would it be better just using DNS names in kestrel configuration?

What would this entail? Does that mean that the server has to nslookup to translate its IP to hostnames? We're unlikely to build something like that in unless other features in .NET need it for something. We don't use the listening address for anything other than binding today (on purpose).

The simple/safe thing to do is have the user configure the value.

If you try to auto-configure it, I think you'll always run into cases that don't work "as expected" for someone/some configuration. Running nslookup yourself and using the results might be the closest you can come, but it sounds pretty complex (you have to list all of the IPs, then look them all up, then figure out which of the many hostnames it might return should be used).

I think it's fine to do your best, just keep in mind that there will always be cases you can't address.

[CarlosLanderas]

@ildoc @tidusjar @Genmutant , based on @rynowak feedback, I suppose the best think you can do is configuring the listening ip / dns name in Kestrel and problem resolved. If you are using a docker image setting the listening address to localhost fixes the problem.

I agree this change "works", but might open new problems in then future while configuring the host ip / dns name is straightforward

[ignatandrei]

@rynowak @CarlosLanderas

What if we use this:

Host = Host.Replace("0.0.0.0", "localhost"); Host = Host.Replace("[::]", "localhost");

?

[NicolasREY69330]

@CarlosLanderas 👍 Could you provide any sample from a docker / docker compose perspective to ensure we do it the right way please ?

[edumserrano]

My workaround for the moment was not by changing the kestrel listening address but by mapping the health checks to absolute URLs using localhost such as:

services.AddHealthChecksUI(setupSettings: setup =>
{
  setup.AddHealthCheckEndpoint("ready", "http://localhost/health-checks/ready");
}

Obviously the above only works if the app is running in port 80, if not you also need to specify the port in the absolute URL.

Not sure if there's something messed up with my docker compose project in Visual Studio but no matter what I tried I couldn't get the proposed solution from @CarlosLanderas to get this to work by keeping relative URLs and doing something like:

webBuilder.UseKestrel(options =>
{
  options.ListenLocalhost(80);
});

I would also appreciate some sample to understand how to run in docker/docker compose.

[NicolasReyDotNet]

@edumserrano @CarlosLanderas Hi, any update on this ?

[edumserrano]

@edumserrano @CarlosLanderas Hi, any update on this ?

I didn't find any solution. I just use an absolute url.

[ignatandrei]

What about my previous suggestion?

[NicolasReyDotNet]

@edumserrano You work with a dockercompose file ? If yes I'm interested in seeing it and you startup.cs file as well, could you share these piece of code here ?

[MikeChristensen]

I ran into this same problem today, and here's the solution I came up with. Basically, I create an extension method on IWebHostBuilder which takes the URLs, normalizes them (replaces 0.0.0.0 or * with localhost) and creates the healthcheck endpoints:

internal static class HealthCheckConfigurationExtensions
{
    /// <summary> Normalize an enumeration of URLs to "localhost" if wildcard bindings are used </summary>
    private static IEnumerable<string> Normalize(this IEnumerable<string> urls) =>
        urls.Select(url => Regex.Replace(url, @"^(?<scheme>https?):\/\/((\*)|(0.0.0.0))(?=[\:\/]|$)", "${scheme}://localhost"));
    public static IWebHostBuilder ConfigureHealthcheck(this IWebHostBuilder host, params string[] urls)
    {
        host.ConfigureServices(serviceBuilder =>
        {
            serviceBuilder
                .AddHealthChecksUI(setupSettings =>
                {
                    var uris = urls.Normalize().Select(uri => new Uri(uri, UriKind.Absolute)).ToArray();
                    var httpEndpoint = uris.FirstOrDefault(uri => uri.Scheme == "http");
                    var httpsEndpoint = uris.FirstOrDefault(uri => uri.Scheme == "https");
                    if (httpEndpoint != null) // Create an HTTP healthcheck endpoint
                    {
                        setupSettings.AddHealthCheckEndpoint("HTTP", new UriBuilder(httpEndpoint.Scheme, httpEndpoint.Host, httpEndpoint.Port, "/health.json").ToString());
                    }
                    if (httpsEndpoint != null) // Create an HTTPS healthcheck endpoint
                    {
                        setupSettings.AddHealthCheckEndpoint("SSL", new UriBuilder(httpsEndpoint.Scheme, httpsEndpoint.Host, httpsEndpoint.Port, "/health.json").ToString());
                    }
                });
        });
        return host;
    }
}

Now, when I build my IWebHost, I can configure the healthchecks as well:

var host = new WebHostBuilder()
    .UseKestrel()
    .UseConfiguration(localConfig)
    .UseContentRoot(Directory.GetCurrentDirectory())
    .ConfigureServices(serviceBuilders)
    .Configure(appBuilders)
    .ConfigureHealthcheck(urls) // <--- Extension method
    .UseUrls(urls);

It would also be possible to combine ConfigureHealthcheck and UseUrls into a single extension method that does both of those things.

In my case, I'm adding a healthcheck for the first HTTP binding and the first HTTPS binding found, but your situation might be different and you can adjust the code.

Hope this helps someone!

[maikebing]

@MikeChristensen Thank you very much，I modified the regular expression ^(?<scheme>https?):\/\/((\*)|(0.0.0.0))(?=[\:\/]|$) ^(?<scheme>https?):\/\/((\+)|(\*)|(0.0.0.0))(?=[\:\/]|$)

  var urls = Environment.GetEnvironmentVariable("ASPNETCORE_URLS").Split(';');
                var uris = urls.Select(url => Regex.Replace(url, @"^(?<scheme>https?):\/\/((\+)|(\*)|(0.0.0.0))(?=[\:\/]|$)", "${scheme}://localhost"))
                                .Select(uri => new Uri(uri, UriKind.Absolute)).ToArray();
                var httpEndpoint = uris.FirstOrDefault(uri => uri.Scheme == "http");
                var httpsEndpoint = uris.FirstOrDefault(uri => uri.Scheme == "https");
                if (httpEndpoint != null) // Create an HTTP healthcheck endpoint
                {
                    setup.AddHealthCheckEndpoint("IoTSharp HTTP", new UriBuilder(httpEndpoint.Scheme, httpEndpoint.Host, httpEndpoint.Port, "/healthz").ToString());
                }
                if (httpsEndpoint != null) // Create an HTTPS healthcheck endpoint
                {
                    setup.AddHealthCheckEndpoint("IoTSharp SSL", new UriBuilder(httpsEndpoint.Scheme, httpsEndpoint.Host, httpsEndpoint.Port, "/healthz").ToString());
                }
                else
                {
                    //One endpoint is configured in appsettings, let's add another one programatically
                    setup.AddHealthCheckEndpoint("IoTSharp", "/healthz");
                }

[MaxThom]

Hi, I've been trying so many things to fix this issues. Ultimately, im trying to deploy on Azure WebApp for Linux Container, but for now i cant get it to work even on my local machine. Everything work except this. I tried settings env variable with urls: "http://localhost:80;https://localhost:443", but then even the index break. I tried to set absolute address on health endpoint, but without success. I changed Kestrel UseUrls to localhost, without success.

Any tips ?

[koimad]

Has anybody resolved this issue, I'm getting the same problems.

[SumitAngra]

Hi Everyone, Facing the same issue for the service fabric cluster. Kindly do let me know if this is resolved or not. Thanks

[michalkonieczny91]

To workaround the problem in containers you have to specify a single IP address for listening instead all To do this you can follow

k8s:

Use downward API to get container IP (MY_POD_IP) using status.podIP, then use this MY_POD_IP into ASPNETCORE_URLS "http://$(MY_POD_IP):80" below piece of manifest

            - name: MY_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: ASPNETCORE_URLS
              value: "http://$(MY_POD_IP):80"

docker-compose:

you can use same trick but instead of using API you can use just linux hostname command

    entrypoint: [ "/bin/sh","-c" ]
    command:
      - |
        sleep 60
        export ASPNETCORE_URLS="http://$$(hostname -i):80;https://$$(hostname -i):443"
        dotnet /app/.web-ui.dll

you have to overide entrypoint to invoke shell script from command and you can find a one liner export ASPNETCORE_URLS="http://$$(hostname -i):80;https://$$(hostname -i):443"

A $$(hostname -i) statement is get container IP and the double $$ is for escape

Hope it will help :)

[RyanSearle]

The whole UI feature is unusable when using Kestrel with a dynamic port.

If HealthChecks.UI.Configuration.Settings allowed me to provide the BaseAddress for the HttpClient with a delegate which supplies the IServiceProvider I could dynamically add the port before each request.

Why does the port need to be known upfront?

BTW it looks like ConfigureApiEndpointHttpclient should do this however changing the BaseAddress does nothing.

Fundamentally doing a replace on 0.0.0.0 to 127.0.0.1 fixes the issue for me:

targetAddress = targetAddress.Replace("0.0.0.0", "127.0.0.1");

[bayramerenn]

i had the same problem. Can you share an example so I can solve it?

[RyanSearle]

I've forked the repo with the required change https://github.com/RyanSearle/AspNetCore.Diagnostics.HealthChecks.

You'll just need to build it and use the package.

Alternatily look at the first commit and make the change yourself.