Traces Not Appearing in Application Insights Despite Sampling Configuration

[jacopocarlini]

Expected behavior

I expect the sampling overrides to work correctly. I want to capture 100% of the traces if the message starts with Invoking.

Actual behavior

The override is ignored.

To Reproduce

I'm using Spring Boot v3.2.3 along with the latest applicationinsights-agent-3.6.0.jar.

Below is my applicationinsights.json configuration:

{
  "selfDiagnostics": {
    "destination": "console",
    "level": "INFO"
  },
  "sampling": {
    "percentage": 0,
    "overrides": [
      {
        "telemetryType": "dependency",
        "percentage": 5
      },
      {
        "telemetryType": "trace",
        "percentage": 100,
        "attributes": [
          {
            "key": "message",
            "value": "Invoking .*",
            "matchType": "regexp"
          }
        ]
      },
      {
        "telemetryType": "exception",
        "percentage": 100
      },
      {
        "telemetryType": "request",
        "percentage": 100
      }
    ]
  }
}

The application produces two log entries:

"log.level": "INFO","message":"Invoking API operation healthCheck",
"log.level": "INFO","message":"Ignore this log"

However, I don't see any traces appearing in Application Insights.

System information

Please provide the following information:

• SDK Version: 3.6.0
• OS type and version: on docker image eclipse-temurin:17-jre
• Application Server type and version (if applicable):
• Using spring-boot? yes
• Additional relevant libraries (with version, if applicable):

Logs

Screenshots

If applicable, add screenshots to help explain your problem.

[jeanbisutti]

@jacopocarlini

The log message is not an available attribute for logs.

You could add a log marker if the log is generated from your application. An example: https://github.com/microsoft/ApplicationInsights-Java/blob/ee1508cf1ee5c9ee758f50f45550dcb485f97a3f/smoke-tests/apps/Logback/src/main/java/com/microsoft/applicationinsights/smoketestapp/LogbackServlet.java#L30

You could after enable the capture of markers as an attribute: https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config#log-markers-preview The attribute name will be logback.marker for Logback and log4j.marker for Log4j.

Another option might be to enable the capture code attributes: https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config#other-log-attributes-for-logback-preview An example of code attributes:

code.filepath="FrameworkServlet.java", code.function="initServletBean", code.lineno=554, code.namespace="org.springframework.web.servlet.FrameworkServlet"

Be aware that capturing code attributes might add a performance overhead.

You can also mask sensitive data of the telemetry log body: https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-telemetry-processors-examples#masking-sensitive-data-in-log-message

An alternative solution might be to use a Data Collection Rule (DCR): https://learn.microsoft.com/en-us/azure/azure-monitor/logs/tutorial-workspace-transformations-portal

[jacopocarlini]

Thank you for your reply, @jeanbisutti.

I’ve updated my Java code as follows:

log.info("Ignore this log info");
Marker marker = MarkerFactory.getMarker("aMarker");
log.info(marker, "Invoking API operation healthCheck");

I’m using the following configuration:

{
  "selfDiagnostics": {
    "destination": "console",
    "level": "INFO"
  },
  "preview": {
    "captureLogbackMarker": true
  },
  "sampling": {
    "percentage": 0,
    "overrides": [
      {
        "telemetryType": "dependency",
        "percentage": 5
      },
      {
        "telemetryType": "trace",
        "percentage": 100,
        "attributes": [
          {
            "key": "logback.marker",
            "value": "aMarker",
            "matchType": "regexp"
          }
        ]
      },
      {
        "telemetryType": "exception",
        "percentage": 100
      },
      {
        "telemetryType": "request",
        "percentage": 100
      }
    ]
  }
}

However, the logs are still not being sent to Application Insights. Could you help me identify what I might be missing?

Additionally, could you share any documentation that includes a list of the default attributes I can use?

Thank you

[jeanbisutti]

@jacopocarlini Unfortunately a log marker attribute can't be used today with the sampling override (array attribute).

[jeanbisutti]

@jacopocarlini Another option could be to add an MDC attribute and use it in the sampling override:

    MDC.put("attributeKey", "attributeValue");
    try {
      logger.info("Log message");
    } finally {
      MDC.remove("attributeKey");
    }

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 7 days. It will be closed if no further activity occurs within 7 days of this comment.