Custom attributes are not populated in Azure Functions' Request spans

[leninalive]

Expected behavior

In Application Insights, custom attributes assigned to Request spans (function invocations) are present.

Actual behavior

Only predefined pre-populated attributes are available as custom dimensions (like TriggerReason, FullName, OperationName etc) for function invocation requests.

Sample Application

public class TestFunctionHandler extends FunctionInvoker<Event, String> {

    @FunctionName("testFunction")
    public void execute(
            @QueueTrigger(name = "event", queueName = "queue",
                          connection = "AzureWebJobsStorage")
            Event event,
            ExecutionContext context) {
        Span.current().setAttribute("TestAttribute", event.getData());
        handleRequest(job, context);
    }
}

This code supposed to produce "TestAttribute" in "Custom Dimensions" panel of App Insights' Request, however, only default attributes are actually populated there:

[heyams]

Azure function host is using dotnet sdk to track requests. Can you raise an issue at ApplicationInsights-dotnet? I will give dotnet SDK Team a heads up.

[leninalive]

Well, I found that with .NET, probably it is possible to add custom properties using OpenTelemetry SDK like this:

Activity.Current.AddTag("Test", "Test");

Not tried it on my own though. Meanwhile, that repository promotes me to create Azure Functions-related issue in another repository:

If you're running the application in Azure Functions, please file an issue here.

Please advice the correct one.

[heyams]

After further discussion within my team, we'll discuss it with Azure Function team. Span.current().setAttribute("TestAttribute", event.getData()); Span.current() is a noop.

In the meantime, can you try @withSpan instead:

@WithSpan
private void handleRequest(job, context, event) { // or pass in event.getData directly
  Span.current().setAttribute("TestAttribute", event.getData()); 
  // + your own implementation
}

in your sample app, it will look like this:

public class TestFunctionHandler extends FunctionInvoker<Event, String> {

    @FunctionName("testFunction")
    public void execute(
            @QueueTrigger(name = "event", queueName = "queue",
                          connection = "AzureWebJobsStorage")
            Event event,
            ExecutionContext context) {
        handleRequest(job, context, event);
    }
}

Please let me know the outcome.

[leninalive]

handleRequest() in my code isn't my own private function but Spring Cloud Function Azure Adapter one. I've tried to wrap underlying TestFunction.apply method with @WithSpan annotation but it creates serious hassle looking it up later in App Insights because I have hundreds of other methods wrapped with this annotation.

My goal is to create a simple way to locate a specific Function call, I have only a few functions in my app though looking them up with Request Name filter is much simpler than plain Name filter.

[leninalive]

In the meantime, you stated that

Span.current() is a noop.

However, I did some checks and found that Span.current()'s SpanId and TraceId are exactly matching corresponding span's properties in App Insights.

I found the following code in agent:

      RpcTraceContext traceContext = request.getTraceContext();
      Context extractedContext =
          GlobalOpenTelemetry.getPropagators()
              .getTextMapPropagator()
              .extract(Context.root(), traceContext, GETTER);
      SpanContext spanContext = Span.fromContext(extractedContext).getSpanContext();

      // recreate SpanContext to override the trace flags since the host currently always sends "00"
      TraceFlags traceFlags =
          BytecodeUtil.shouldSample(spanContext.getTraceId())
              ? TraceFlags.getSampled()
              : TraceFlags.getDefault();
      spanContext =
          SpanContext.createFromRemoteParent(
              spanContext.getTraceId(),
              spanContext.getSpanId(),
              traceFlags,
              spanContext.getTraceState());

      Map<String, String> attributesMap = traceContext.getAttributesMap();
      AzureFunctionsCustomDimensions customDimensions =
          new AzureFunctionsCustomDimensions(
              request.getInvocationId(),
              attributesMap.get("ProcessId"),
              attributesMap.get("LogLevel"),
              attributesMap.get("Category"),
              attributesMap.get("HostInstanceId"),
              attributesMap.get("#AzFuncLiveLogsSessionId"));

      return Context.current().with(Span.wrap(spanContext)).with(customDimensions).makeCurrent();

But not sure how it works exactly and what is it for.

[trask]

hi @leninalive! the request span is captured on the Azure Functions host. In the Java worker, we set up Span.current() with the same traceId/spanId, so that dependency spans which are captured on the Azure Functions worker will parent the request span.

but setting attributes onto that Span.current() aren't currently sent back from the worker to the host where they could be added to the request span.

[leninalive]

Hi @trask yes, it became clear to me right after I submitted that comment and digged a little more.

So far I see that Span.current() will be a PropagatedSpan, though even inheritedAttributes agent's feature won't work that way.

Seems I have only to wait for this feature to be implemented later if it ever will.