aws-otel-java-instrumentation icon indicating copy to clipboard operation
aws-otel-java-instrumentation copied to clipboard

feat: [Java] EMF Exporter Implementation

Open liustve opened this issue 2 months ago • 0 comments

Description of changes: Java Version of these PRs:

  • https://github.com/aws-observability/aws-otel-python-instrumentation/pull/382
  • https://github.com/aws-observability/aws-otel-python-instrumentation/pull/409
  • https://github.com/aws-observability/aws-otel-python-instrumentation/pull/410
  • https://github.com/aws-observability/aws-otel-python-instrumentation/pull/434

This PR introduces the complete CloudWatch EMF (Embedded Metric Format) exporter implementation for sending OpenTelemetry metrics directly to CloudWatch without requiring a Collector or Agent.

In order to enable this exporter, users MUST set the following environment variables:

  • OTEL_METRICS_EXPORTER=awsemf -
  • OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=<log-group-name>,x-aws-log-stream=<log-stream-name>, x-aws-metric-namespace=<namespace>
  • AWS_REGION=<region> OR AWS_DEFAULT_REGION=<region>

This PR includes:

  • EMF MetricRecord translation for for unified representation of all OTel metric types with log creation with unit mapping from OpenTelemetry to CloudWatch-compatible units
  • Automatic log group and stream creation with retry logic
  • Supported CloudWatch Logs integration with batching and constraint handling (256KB event limit, 1MB request limit, timestamp limits)
  • Support for metric grouping by attributes and timestamps for EMF log generation
  • Support for Gauge, Sum, Histogram, and ExponentialHistogram metric types

TODO:

  • On the next PR, will integrate the Console EMF exporter into AWS Lambda environments to validate EMF log formatting and ensure consistent behavior in Lambda runtime: https://github.com/aws-observability/aws-otel-python-instrumentation/pull/437

Testing:

  • Added unit tests to validate EMF exporter configuration scenarios, including parameterized tests for both valid configurations (supporting AWS_REGION and AWS_DEFAULT_REGION) and invalid configurations (missing headers, wrong exporter type, missing region). The tests ensure the EMF exporter is correctly enabled only when all required environment variables are properly configured.

  • Manual end to end testing with the following environment variables to ensure the EMF logs show up:

    • AWS_REGION=us-east-1
    • OTEL_METRICS_EXPORTER=awsemf
    • OTEL_EXPORTER_OTLP_LOGS_HEADERS=x-aws-log-group=test,x-aws-log-stream=default,x-aws-metric-namespace=testNamespace
    • OTEL_RESOURCE_ATTRIBUTES=service.name=testService,aws.log.group.names=test,cloud.resource_id=agent-12345
    • OTEL_LOGS_EXPORTER=none
    • OTEL_TRACES_EXPORTER=none

Example EMF log emitted:

{
    "otel.resource.process.command_args": "[/Library/Java/JavaVirtualMachines/amazon-corretto-21.jdk/Contents/Home/bin/java, -javaagent:/Users/liustve/aws-otel-java-instrumentation/otelagent/build/libs/aws-opentelemetry-agent-2.18.0-SNAPSHOT.jar, -Dfile.encoding=UTF-8, -Dsun.stdout.encoding=UTF-8, -Dsun.stderr.encoding=UTF-8, -jar, /Users/liustve/aws-otel-java-instrumentation/sample-apps/springboot/build/libs/springboot-2.11.0-SNAPSHOT.jar]",
    "otel.resource.host.arch": "aarch64",
    "otel.resource.host.name": "7cf34dd812df",
    "otel.resource.service.instance.id": "a0399d3c-b856-43ae-b374-fe66dee41ce8",
    "otel.resource.aws.log.group.names": "test",
    "jvm.class.unloaded": 1,
    "otel.resource.service.name": "testSErvice",
    "_aws": {
        "CloudWatchMetrics": [
            {
                "Metrics": [
                    {
                        "Unit": "Count",
                        "Name": "jvm.cpu.count"
                    },
                    {
                        "Unit": "Count",
                        "Name": "jvm.class.loaded"
                    },
                    {
                        "Name": "jvm.cpu.recent_utilization"
                    },
                    {
                        "Unit": "Seconds",
                        "Name": "jvm.cpu.time"
                    },
                    {
                        "Unit": "Count",
                        "Name": "jvm.class.count"
                    },
                    {
                        "Unit": "Count",
                        "Name": "jvm.class.unloaded"
                    }
                ],
                "Namespace": "testNamespace"
            }
        ],
        "Timestamp": 1758761023497
    },
    "otel.resource.cloud.resource_id": "agent-12345",
    "jvm.class.count": 14264,
    "Version": "1",
    "otel.resource.process.pid": "46822",
    "otel.resource.os.description": "Mac OS X 15.6.1",
    "otel.resource.telemetry.distro.name": "opentelemetry-java-instrumentation",
    "otel.resource.os.type": "darwin",
    "otel.resource.telemetry.sdk.name": "opentelemetry",
    "otel.resource.telemetry.distro.version": "2.18.0-aws-SNAPSHOT",
    "otel.resource.process.runtime.description": "Amazon.com Inc. OpenJDK 64-Bit Server VM 21.0.8+9-LTS",
    "otel.resource.process.runtime.version": "21.0.8+9-LTS",
    "jvm.cpu.recent_utilization": 0,
    "otel.resource.process.executable.path": "/Library/Java/JavaVirtualMachines/amazon-corretto-21.jdk/Contents/Home/bin/java",
    "otel.resource.telemetry.sdk.version": "1.52.0",
    "jvm.cpu.count": 12,
    "jvm.class.loaded": 14265,
    "otel.resource.process.runtime.name": "OpenJDK Runtime Environment",
    "otel.resource.telemetry.sdk.language": "java",
    "jvm.cpu.time": 7.756965
}

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

liustve avatar Sep 19 '25 22:09 liustve