azure-functions-host icon indicating copy to clipboard operation
azure-functions-host copied to clipboard

Health Checks - Timer trigger executed on unhealthy instance

Open asierpn opened this issue 1 year ago • 8 comments

We have implemented Health Check in our Functions Apps deployed in a dedicated plan, if one of the instances is unhealthy, the functions host still redirects timer trigger functions to the unhealthy instance, while all calls to http trigger functions are correctly redirected to the healthy instances.

Repro steps

Enable Health Check and use a Timer Trigger and an Http Trigger function with 2 or more instances, forcing one of them to be unhealthy in the Health Check function implementation.

Expected behavior

The Timer Trigger function is redirected to one healthy instance.

Actual behavior

The Timer Tigger function sometimes is redirected to the unhealthy instance.

asierpn avatar Aug 16 '24 08:08 asierpn

Hi @asierpn thanks for reporting please share the all repro steps. and func app name,invocation id ,timestamp region where you deployed if.

bhagyshricompany avatar Aug 19 '24 11:08 bhagyshricompany

Here are the details:

Invocation Id: 367a988c-b426-4eb5-995b-00f9026b5119 Timestamp: 2024-08-16T08:50:00.0002035Z Region: North Europe

Repro steps:

Create two functions, one for the health check, which is hacked to return an unhealthy state on one of the two deployed instances:

    [Function("HealthCheck")]
    public async Task<IActionResult> HealthCheck([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "health")] HttpRequest req)
    {
      if (Environment.GetEnvironmentVariable("COMPUTERNAME") == "XXX")
      {
        return new ObjectResult(Enum.GetName(typeof(HealthStatus), HealthStatus.Unhealthy))
        {
          StatusCode = (int?)HttpStatusCode.ServiceUnavailable
        };
      }
      return new OkObjectResult(Enum.GetName(typeof(HealthStatus), HealthStatus.Healthy));
    }

And another one for the timer trigger:

    [Function($"TestFunction")]
    public void TestFunction([TimerTrigger("0 */1 * * * *")] TimerInfo timer)
    {
      Logger.LogInformation($"Test executed in {Environment.GetEnvironmentVariable("COMPUTERNAME")}");
    }

Deploy across at least two instances (we use Linux Containers), enable the health check in the Function App and check the logs, the TestFunction should only be redirected to the healthy instance.

asierpn avatar Aug 23 '24 05:08 asierpn

Is there any update on this issue at all?

asierpn avatar Sep 12 '24 06:09 asierpn

will checking update you soon.Thanks

bhagyshricompany avatar Sep 17 '24 13:09 bhagyshricompany

please share your log information details and invocation id,timestamp region etc.Thanks

bhagyshricompany avatar Oct 17 '24 12:10 bhagyshricompany

please share your log information details and invocation id,timestamp region etc.Thanks

I shared it on 23 August in this comment: https://github.com/Azure/azure-functions-host/issues/10403#issuecomment-2306340618 ....

asierpn avatar Oct 17 '24 12:10 asierpn

can you share which doc you refered?

bhagyshricompany avatar Oct 22 '24 14:10 bhagyshricompany

https://learn.microsoft.com/en-us/azure/azure-functions/configure-monitoring?tabs=v2#monitor-function-apps-using-health-check

asierpn avatar Oct 22 '24 14:10 asierpn

@kshyju please comment and validate.same.Thanks

bhagyshricompany avatar Nov 05 '24 15:11 bhagyshricompany