Support structured logging from worker

[brettsam]

The .NET worker is getting requests for structured logging from the worker -> host -> app insights. We'll need to update the GRPC contract for this to work.

[brettsam]

Related: https://github.com/Azure/azure-functions-dotnet-worker/issues/423

[fabiocav]

Moving this to 108 as we're still working with other teams to get feedback and work through the design

[kjeske]

Could you provide us with some time estimates for this fix? It's blocking https://github.com/Azure/azure-functions-dotnet-worker/issues/423 and it seems this bug is being constantly postponed.

[fabiocav]

@kjeske we're still working on some design/open questions and the issue is primarily tracking that work at the moment. When we have more details and start on implementation, we'll be in a better position to provide an ETA

[tommck]

Still kicking this can down the road it seems :/

[emmorts]

@fabiocav, any update on this?

This is quite a crucial feature in our use case. Is there an alternative you could suggest for the time being?

[kelps]

As a workaround, I thought of adding a custom logger factory to DI that would create an ILogger that doesn't flow through the host GRPC channel and would connect directly to Insights. In theory it should work, but we would loose all console and stream logs in Azure and local dev and would be forced to rely on Insights for ANY diagnostics. If we try to keep the connection to GRPC, to also show in the stream logs, Insights would get duplicated entries with and without the structured information.

It should work, but I didn't have the time to try and test it yet.

This NEEDS to be FIXED soon. It is a very serious bug. From the looks of it, I'm worried that it might only happen for .NET 7, but that would be a serious case of not giving enough importance to very important community feedback. This specific issue is already 7 months old, and it was not the first on this problem.

If the intention really is to have only isolated process in .NET 7, as the roadmap suggests, EVERYTHING that work in process will need to work in isolated, including structured logs.

Please make it available as a fix in 6, not only in 7!

[PhilippK13]

Are there any updates on this? We use structured logging heavily in our functions and were seriously considering changing to isolated process but it looks like this issue will be a blocker for us.

[pzaj2]

Any updates on this? This is a major blocker for many people depending on working structured logging 👀

[timmkrause]

@pzaj2 I gave up on this. Hope it will be fixed in .NET 7.

[tommck]

I'm thinking .NET 8 at the earliest

[marios-siatis]

Hey guys I have a workaround for structured logging by using serilog, to replace the existing ms logger. I think is clean as I only need to modify the startup of the application.

https://medium.com/@marios.shiatis/addressing-the-structured-logging-application-insights-issues-in-azure-functions-v4-net-6-0-f1f63b99807d

[colinrippeyfinarne]

@fabiocav @brettsam is there any update you can share on this issue? I am conscious that as we approach the release of dotnet 7 the capability to go "all in" on the isolated model will be pushed (in particular the support for durable functions which has made me stay with the in-proc model for now) but not if structured logging is still not working?

[timmkrause]

At least an "official" statement regarding the roadmap of this issue would be something.

Transparency is key.

[cloudmelon]

hi @colinrippeyfinarne the durable function support is currently in public preview.

https://github.com/Azure/azure-functions-dotnet-worker/projects/2

cc @cgillum

[cloudmelon]

Had a quick check, we had done the work to pull AppInsights logs directly from Worker -> AppInsights, so it supports anything .NET supports directly

Related issue : https://github.com/Azure/azure-functions-dotnet-worker/pull/944#issue-1282987627

Feel free to try it out : https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.ApplicationInsights/1.0.0-preview1

[MullenStudio]

Had a quick check, we had done the work to pull AppInsights logs directly from Worker -> AppInsights, so it supports anything .NET supports directly

Related issue : Azure/azure-functions-dotnet-worker#944 (comment)

Feel free to try it out : https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.ApplicationInsights/1.0.0-preview1

I just tested and it seems that it still doesn't support structured log.

I noticed this line in the code options.IncludeScopes = false;

But even if I set IncludeScopes to true in the options, it still doesn't work.

builder.AddApplicationInsights().AddApplicationInsightsLogger(options =>
                    {
                        options.IncludeScopes = true;
                    });

Any suggestion?

[arkiaconsulting]

Tested with https://www.nuget.org/packages/Microsoft.Azure.Functions.Worker.ApplicationInsights/1.0.0-preview3. Still no scoped properties.

[mxvoloshin]

Any plans when this will be working?

[madushans]

This is broken and prevents using .NET 7 and having any meaningfully useful logs and metrics.

PLEASE consider updating execution mode comparison to prevent people wasting their time upgrading and end up breaking logs and dashboards that rely on structured logging and custom metrics.

[sid2934]

Do we have an idea of when this work may be picked up?

[madushans]

Unfortunately doesn't look like it. The team appears to be focusing on other thinks like Durable Functions and updates to Python experience.

Since worker/isolated mode is the way forward for dotnet as well, this will have to be fixed be ready before dotnet 6 goes out of support (12 Nov 2024). Until then, dotnet 6 is supported in the in-process mode which has this working, and it is LTS.

Bit less pessimistic view is they may fix this before end of 2023 (or mid 2024 like they did for dotnet 7 in mid 2022?) so people can move to dotnet 8 LTS in time to get out of dotnet 6.

[DOMZE]

I was able, through the other issue, to get structured logging to show up in application insights.

Here's a snippet on how to get this going. Note that I deployed this on a Windows App Service Plan. Did not try on Linux. Install the Microsoft.Extensions.Logging.ApplicationInsights package to get the extension method to be discoverable on the ILoggingBuilder

class Program
{
	static async Task Main(string[] args)
	{
		var host = new HostBuilder()
			.ConfigureFunctionsWorkerDefaults()
			.ConfigureLogging((context, builder) =>
			{
				// does not work as per https://github.com/Azure/azure-functions-dotnet-worker/issues/423
				//builder.AddApplicationInsights();

				var applicationInsightsConnectionString = context.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];
				if (!string.IsNullOrEmpty(applicationInsightsConnectionString))
				{
					builder.AddApplicationInsights(configureTelemetryConfiguration =>
					{
						configureTelemetryConfiguration.ConnectionString = applicationInsightsConnectionString;
					}, _ => { });
				}
			})
			.Build();

		await host.RunAsync();
	}
}

[brettsam]

This is the right direction @DOMZE. We have a specific package in preview right now that wraps the default App Insights packages and does this wire-up for you. There's a couple small tweaks it makes, but behind-the-scenes it's basically doing what you're doing there.

See this for more details on the App Insights package (including the "docs" in the PR): https://github.com/Azure/azure-functions-dotnet-worker/issues/1263

[JayAtSherweb]

@brettsam Your documentation seems to outline how to use in an application, but not an isolated function. Please help.

[brettsam]

@JayAtSherweb -- Once you've set things up with App Insights as mentioned in the docs (specifically, calling AddApplicationInsightsLogger(), any logging you do will go directly to Application Insights. Their ILogger implementation supports structured logging. I couldn't find an official doc with an example, but a log like this:

_logger.LogWarning("C# HTTP trigger function processed a request. {guid}", Guid.NewGuid());

will write the message to the traces table in App Insights -- and you'll see the guid extracted into the customDimensions.

[p-m-j]

@brettsam - This doesn't seem to work with ILogger.BeginScope<TState> however

e.g.

  [Function("Pong")]
  public void QueueTrigger([QueueTrigger("ping")] string myQueueItem)
  {
      using var scope = _logger.BeginScope(new Dictionary<string, object>{ ["A"] = "12345" });

      _logger.LogInformation("pong {B}", "12345");
  }

Has a customDimension.B but no customDimension.A present

Edit: By this I mean

<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.0.0-preview3" />

[brettsam]

@p-m-j -- can you make sure to set up App Insights to IncludeScopes for now? This is incorrectly set to false in the last preview -- but I'm removing this line in a PR soon:

https://github.com/Azure/azure-functions-dotnet-worker/blob/584cab006b82e0cbc2b95a8b66e2546d3a8ddfe4/src/DotNetWorker.ApplicationInsights/FunctionsApplicationInsightsExtensions.cs#L51

You can override this with

.AddApplicationInsightsLogger(o => o.IncludeScopes = true);

[p-m-j]

@brettsam works like a charm, cheers

[fleed]

I was able, through the other issue, to get structured logging to show up in application insights.

Here's a snippet on how to get this going. Note that I deployed this on a Windows App Service Plan. Did not try on Linux. Install the Microsoft.Extensions.Logging.ApplicationInsights package to get the extension method to be discoverable on the ILoggingBuilder

class Program
{
	static async Task Main(string[] args)
	{
		var host = new HostBuilder()
			.ConfigureFunctionsWorkerDefaults()
			.ConfigureLogging((context, builder) =>
			{
				// does not work as per https://github.com/Azure/azure-functions-dotnet-worker/issues/423
				//builder.AddApplicationInsights();

				var applicationInsightsConnectionString = context.Configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];
				if (!string.IsNullOrEmpty(applicationInsightsConnectionString))
				{
					builder.AddApplicationInsights(configureTelemetryConfiguration =>
					{
						configureTelemetryConfiguration.ConnectionString = applicationInsightsConnectionString;
					}, _ => { });
				}
			})
			.Build();

		await host.RunAsync();
	}
}

Thank you @DOMZE ! But how do you configure the level for a specific category? I tried the "standard" way, but it doesn't seem to work:

host.json:

{
    "version": "2.0",
    "logging": {
        "applicationInsights": {
            "samplingSettings": {
                "isEnabled": true,
                "excludedTypes": "Request"
            }
        },
        "logLevel": {
            "MyNamespace.MyClass": "Trace"
        }
    }
}

The logger MyNamespace.MyClass doesn't send anything below Information.

I get logs if I manually set the log level in code:

builder.SetMinimumLevel(LogLevel.Trace);

@brettsam from my tests, it also works if I add the package Microsoft.Azure.Functions.Worker.ApplicationInsights 1.0.0-preview4 and avoid the call builder.AddApplicationInsights(...), but still the configuration level is not set for a specific logger<T> (at least not via host.json).