Add OTLP stdout exporter

[zeitlinger]

Towards https://github.com/open-telemetry/opentelemetry-specification/issues/3817

Changes

Add OTLP Stdout exporter.

For non-trivial changes, follow the change proposal process.

• [x] Related issues https://github.com/open-telemetry/opentelemetry-specification/issues/3817 ~* [ ] Related OTEP(s) #~
• [x] Links to the prototypes: https://github.com/open-telemetry/opentelemetry-java/pull/6632
• [x] CHANGELOG.md file updated for non-trivial changes ~* [ ] spec-compliance-matrix.md updated if necessary~
• [ ] reference implementation https://github.com/open-telemetry/opentelemetry-java/pull/6632

[pellared]

What about standard error and and file as output destinations?

[zeitlinger]

How does this relate to the existing console/stdout and file exporters? This seems at best overlapping, but at worst duplicative of already specified exporters.

• console/stdout: is for debugging purposes - it doesn't have a specified format, which you need for https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/otlpjsonconnector
• file: it is exactly the stdout part of that, I failed to link that. in the spec meeting, we discussed that it would be easier to start with stdout as a subset to keep it simple

[zeitlinger]

@pellared @MrAlias can you check again?

[cijothomas]

Fixes https://github.com/open-telemetry/opentelemetry-specification/issues/3817

The linked issue is about OTLP File exporter, but this seems to talking about stdout exporter... Did I misread this?

[zeitlinger]

Fixes #3817

The linked issue is about OTLP File exporter, but this seems to talking about stdout exporter... Did I misread this?

otlp-stdout is part of the file exporter

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[zeitlinger]

@pellared @MrAlias can you check again?

[zeitlinger]

@cijothomas can you check again?

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[zeitlinger]

still working on the pr

[cyrille-leclerc]

As discussed with @zeitlinger, this capability to export standardized OTel logs in files that can be seamlessly ingested by the OTel Collector is a major improvement for OTel logs.

That being said, I would be interested in understanding better the developer experience, how this OTEL_LOGS_EXPORTER=otlp/stdout config option will co-exist with the config of logging framework like Logback in Java?

Taking the example of a Java Spring Boot developer, I'll typically use the default logs configuration that emits unstructured logs in stdout (example below) and, if I want to modify this, I'll modify this config through Spring's application.properties and logback-spring.xml files (see reference docs here).

If I want to modify the stdout output of my Spring Boot application to adopt OTel JSON logging to replace the unstructured logs,

1. I'll expect to have to modify these Spring Boot config files for logging: application.properties and logback-spring.xml
2. I'll have to modify these Spring Boot config files to disable the unstructured log messages that are outputted in stdout.

I think that Elastic has done this very well with the Elastic ECS Logging framework. In addition to a manual config, Elastic has an experimental config flag log_ecs_reformatting do seamlessly change the logger config and disable the pre-existing config in favor of the ECS/JSON one. Maybe @AlexanderWert could share some lessons learned.

I'm wondering if we could start with implementing a few extensions of popular logging frameworks (LogBack, Log4j, Pino...) that developers would manually use in their logs config.This could be a natural extension of the OTLP gRPC and HTTP appenders we have implemented so far (eg [here](Appender Instrumentation for Logback version 1.0 and higher ))

I would imagine to the following developer experience for Spring Boot app:

• export OTEL_LOGS_EXPORTER=none to disable the otlp http output
• Edit logback-spring.xml to:

<configuration>
  <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
     <!-- below is just an idea, don't copy paste this fragment -->
     <encoder class="io.opentelemetry.....OtlpJsonEncoder">
  </appender>
  ...
  <root level="INFO">
    <appender-ref ref="console"/>
  </root>

</configuration>

Example Spring Boot console logs (Spring docs here):

2024-08-22T16:59:59.387Z  INFO 109360 --- [myapp] [           main] o.s.b.d.f.logexample.MyApplication       : Starting MyApplication using Java 17.0.12 with PID 109360 (/opt/apps/myapp.jar started by myuser in /opt/apps/)
2024-08-22T16:59:59.410Z  INFO 109360 --- [myapp] [           main] o.s.b.d.f.logexample.MyApplication       : No active profile set, falling back to 1 default profile: "default"
2024-08-22T17:00:02.105Z  INFO 109360 --- [myapp] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
2024-08-22T17:00:02.133Z  INFO 109360 --- [myapp] [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-08-22T17:00:02.133Z  INFO 109360 --- [myapp] [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.28]
2024-08-22T17:00:02.296Z  INFO 109360 --- [myapp] [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-08-22T17:00:02.301Z  INFO 109360 --- [myapp] [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2647 ms
2024-08-22T17:00:03.219Z  INFO 109360 --- [myapp] [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path '/'
2024-08-22T17:00:03.250Z  INFO 109360 --- [myapp] [           main] o.s.b.d.f.logexample.MyApplication       : Started MyApplication in 5.793 seconds (process running for 7.164)

[martinjt]

I feel this is a bad direction for OpenTelemetry in general.

stdout for debugging is a valid usecase, but we should strongly encourage people to use OTLP and push the data to an endpoint (ideally a collector). We should not be promoting push data to the console/stdout and using another tool to pull that data and ingest it. I understand that K8s has done it this way in the past, and that 12 factor apps popularised this approach, but now we have a standardised protocol and we don't need that now.

From inefficiencies in some languages, to issues with polling for that data, it's just not something that we should push for when we have a lot of alternatives like OTLP and collector, that are part of the OpenTelemetry project.

All the requests I've seen from this in the .NET world have been from people not wanting to move from other technologies like fluent etc. and not that it's a better way to work.

I can understand a desire to have a console logging output that formats the log line with the template to produce the human readable string, but not one who's purpose is to provide a format for other tools to use. This will fragment the ecosystem further than it already is.

Right now, we have a path that we push people down. Use the OTLP exporter to push data to an endpoint that accepts OTLP. If we push down this route as a production usecase, that will become diluted with vendors saying "use our agent and configure OpenTelemetry to push to stdout", and still claim that they support OpenTelemetry. Especially if this is then supported for Metrics and Traces too. That will actively harm OpenTelemetry as a project, when we're trying make the developer experience better.

[pellared]

@martinjt, what about people following https://12factor.net/logs?

A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout.

For some reasons https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md was added to OpenTelemetry.

[martinjt]

@pellared the 12fa methodology was created in a different time. I've blogged about this https://www.honeycomb.io/blog/twelve-factor-apps-modern-observability

The general idea being that "apps shouldn't care about where the logs are going"... and if you're using OpenTelemetry, that's fine because we have the OTEL_EXPORTER_OTLP_ENDPOINT.

Pushing to a file is a very different idea, and has different usecases like offline access. That's different to trying to replicate realtime egress of the data using other tools.

[pellared]

Pushing to a file is a very different idea, and has different usecases like offline access

I agree with you. However, notice that sending to stdout is not against this use-case as there are tools (e.g. systemd) that redirect standard output to files.

[martinjt]

@pellared We already have file exporting, that's the otlp file exporter. This is about stdout rather than files. If someone want's file we support that, just not via piping it to stdout in utf8 first, for them to then stream into a file.

[pellared]

@pellared We already have file exporting, that's the otlp file exporter. This is about stdout rather than files. If someone want's file we support that, just not via piping it to stdout in utf8 first, for them to then stream into a file.

See https://github.com/open-telemetry/opentelemetry-specification/pull/4183#issuecomment-2277466995

the spec meeting, we discussed that it would be easier to start with stdout as a subset to keep it simple

[martinjt]

I'm not sure what that comment means exactly? It seems to be suggesting that stdout, with a standard format, is the approach that's being suggested, and I'm saying that it's a mistake, stdout should be for debugging, and format shouldn't need to be consistent. Consistency is only important when it comes to tooling, therefore that would be the only reason to add it to the spec.

[austinlparker]

• Wherever this exporter is documented, we indicate that it is for legacy support only.

I would maybe rephrase it to 'situations where OTLP push is infeasible' rather than 'legacy'. For instance, serverless platforms like Lambda have a fairly robust 'take console output and write it somewhere', I can see this being very useful for those situations. While I'd love to wave a magic wand and have everyone just use OTLP push, I think stuff like that will take a while, so this seems like a good solution until then.

[tedsuo]

That's fair. The warning on the exporter doesn't need to be judgmental, it just needs to prevent users from installing it by mistake.

[felixbarny]

I think that both sending logs out directly via OTLP and logging to a file/stdout have their tradeoffs and use case. OTel should provide an answer for the most common approaches rather than being too prescriptive of what's the right, and what's the wrong approach. OTel should stand both for innovation and supporting tried and tested ingestion paths, to enable a wide range of users to adopt OTel. Startups, big enterprises, greenfield and legacy applications alike.

We've heard a lot about the downsides for log/stdout-based logging architectures. An advantage is that it's usually more resilient, and can sustain a longer period of time without being connected to the backend (like an OTel collector running in a daemonset), without data loss. That's a trait that's appreciated by users that require stricter delivery guarantees for their logs.

A use case that was already alluded to is k8s. While you can also send your logs directly via OTLP in a k8s environment, chances are that not all of your applications are instrumented with OTel. If you want to have a single consistent mechanism how you collect logs from all your k8s pods, I think it'll be useful for OTel to provide a way to log to stdout in a way that will be easy to parse to LogRecords via a standard collector processor/connector.

I do agree with @cyrille-leclerc that ideally, there should be a way to chose an OTLP (like) output format for standard logging libraries. I'd love to find a way to evolve Elastic's ECS loggers into something that works well for OTel. With OTel being an open standard, I think a good approach could be to convince logging library authors to implement an output format that's optimized for OTel. That way, there's no need for an indirection via the logging bridges, and users can leverage the native configuration for their logging library to configure the built-in OTel formatter.

While OTLP JSON would work decently well, an aspect of the ECS loggers is that it aims to be human-readable. While not a must, it would be a great trait to have for an OTel-native logging format.

[cyrille-leclerc]

Thanks @felixbarny, it's a pleasure even though not surprising that we are aligned.

We've heard a lot about the downsides for log/stdout-based logging architectures. An advantage is that it's usually more resilient, and can sustain a longer period of time without being connected to the backend (like an OTel collector running in a daemonset), without data loss.

💯 on all your points regarding the usage of file/stdout over gRPC/HTTP. I would add the following:

• Not all OTel SDKs have production grade support for logs forcing teams to look at alternatives like files and stdout (e.g. "development" status for JavaScript/Node.js, Python, Ruby, Swift, and Erlang)
• The reliability of the gRPC & HTTP OTLP logs pipeline is not meeting yet the requirements of some organizations, particularly due to elements of the OTel Collector:
  • No stable solution to add persistence / storage in the OTel Collector (solutions here are at best in "beta" stage)
  • OTel Collector being perceived as difficult to be used as a reliable / highly-available logs forwarding component. syslog-ng and rsyslog have set the bar very high in terms of reliability.
    • One point that strikes me is that I see a tension between the feature scope and the reliability of the OTel Collector and I'm wondering if a layered architecture with a simplified frontend layer just dedicated to reliable ingestion of telemetry could be relevant.

These elements go beyond this OTEP but I see them as connected.

While OTLP JSON would work decently well, an aspect of the ECS loggers is that it aims to be human-readable. While not a must, it would be a great trait to have for an OTel-native logging format.

💯 if JSON will never be as human readable as plain text logs optimized for humans, an interesting idea for JSON serialization is to sort the json elements on the value for humans rather than on the alphabetical order.

Here is an example of ECS which IMO dos a great job of sorting JSON elements by interest for human rather than relying on alpha orting:

ECS library output with pretty good human readability:

{"@timestamp":"2019-08-06T12:09:12.375Z", "log.level": "INFO", "message":"Tomcat started on port(s): 8080 (http) with context path ''", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.boot.web.embedded.tomcat.TomcatWebServer"}
{"@timestamp":"2019-08-06T12:09:12.379Z", "log.level": "INFO", "message":"Started PetClinicApplication in 7.095 seconds (JVM running for 9.082)", "service.name":"spring-petclinic","process.thread.name":"restartedMain","log.logger":"org.springframework.samples.petclinic.PetClinicApplication"}
{"@timestamp":"2019-08-06T14:08:40.199Z", "log.level":"DEBUG", "message":"init find form", "service.name":"spring-petclinic","process.thread.name":"http-nio-8080-exec-8","log.logger":"org.springframework.samples.petclinic.owner.OwnerController","transaction.id":"28b7fb8d5aba51f1","trace.id":"2869b25b5469590610fea49ac04af7da"}

Equivalent JSON output alpha sorting element with very inferior human readability:

{"@timestamp":"2019-08-06T12:09:12.375Z", "log.level":"INFO", "log.logger":"org.springframework.boot.web.embedded.tomcat.TomcatWebServer", "message":"Tomcat started on port(s): 8080 (http) with context path ''", "process.thread.name":"restartedMain", "service.name":"spring-petclinic" },
{"@timestamp":"2019-08-06T12:09:12.379Z", "log.level":"INFO", "log.logger":"org.springframework.samples.petclinic.PetClinicApplication", "message":"Started PetClinicApplication in 7.095 seconds (JVM running for 9.082)", "process.thread.name":"restartedMain", "service.name":"spring-petclinic" },
{"@timestamp":"2019-08-06T14:08:40.199Z", "log.level":"DEBUG", "log.logger":"org.springframework.samples.petclinic.owner.OwnerController", "message":"init find form", "process.thread.name":"http-nio-8080-exec-8", "service.name":"spring-petclinic", "trace.id":"2869b25b5469590610fea49ac04af7da", "transaction.id":"28b7fb8d5aba51f1" }

[alessandrobologna]

I would maybe rephrase it to 'situations where OTLP push is infeasible' rather than 'legacy'. For instance, serverless platforms like Lambda have a fairly robust 'take console output and write it somewhere', I can see this being very useful for those situations. While I'd love to wave a magic wand and have everyone just use OTLP push, I think stuff like that will take a while, so this seems like a good solution until then.

Completely agree with the above: right now adopting OTEL on serverless (at least AWS Lambda) adds a consistent overhead, both in terms of extra latency in cold starts and execution time, and also memory requirements, if using the ADOT layer. Buffering is not very viable because of the lambda lifecycle, with the execution suspended at the end of the invocation, without a corresponding increase in billed execution time. Finally, limiting the exporting options only to network protocols also requires exposing the collector to the internet, for a non VPC connected lambda, with the extra headaches that come with that. Having a mechanism that would allow to consistently export telemetry to stdout (and so to cloudwatch logs), and ingest them in the collector (possibly using the live tail API, or polling) would be ideal.

[cijothomas]

We see occasional asks about the capability to export to stdout with well defined format. I see no harm in making this part of the spec.

eg: https://github.com/open-telemetry/opentelemetry-dotnet/issues/5844 https://github.com/open-telemetry/opentelemetry-dotnet/issues/5036, https://github.com/open-telemetry/opentelemetry-dotnet/issues/5036#issuecomment-1849734646 https://github.com/open-telemetry/opentelemetry-dotnet/pull/4548#discussion_r1221871214

[pellared]

https://github.com/open-telemetry/opentelemetry-specification/issues/3817 status has been changed to accepted by @open-telemetry/technical-committee based on community feedback. Therefore, I think we can continue reviewing this PR.

[joshhayes-sheen-vt]

Chiming in that this would be helpful for onboarding serverless/lambda use cases incrementally, I'd like to instrument my lambda deployments, then add https://github.com/open-telemetry/opentelemetry-lambda but continue routing my telemetry through cloudwatch (which in that environment is stdout) until I have the rest of my infrastructure deployed and tested, then redirect that stripped down collector to other otlp targets (collectors in k8s/ecs/ec2/whatever) for a more advanced processing pipeline

[cijothomas]

@zeitlinger It'd be great to capture some of the "why" comments from this PR into https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md#opentelemetry-protocol-file-exporter itself for an easy reference in future, "why we need a spec for non OTLP exporter".

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[zeitlinger]

please don't close - just don't have time atm

[github-actions[bot]]

This PR was marked stale due to lack of activity. It will be closed in 7 days.