Allow exporting logs using OTLP exporter

[kushal-ti]

trafficstars

Is your feature request related to a problem? Please describe.

Exporting logs from the application to otel collector directly using an otlp exporter is not possible

A clear and concise description of what the problem is.

It is possible to export traces and metrics to an otel collector, from which they can then be processed and further exported to other sources as required. However, the same is not possible with logs. When logs are attempted to be sent over HTTP to the otel /v1/logs/ endpoint, it returns a 400 bad request error because the logs are not formatted in the OTLP format

Describe the solution you'd like

Integrations into common loggers like winston that would allow exporting the logs into OTLP format, similar to traces exporter and metrics exporter, we could have a logs exported

Describe alternatives you've considered

1. Changing logging configuration to structure logs in the required OTLP format and then sending them to the collector over http
2. Using a middle man like fluent-bit to collect logs from stdout and send them over to the otel collector Both approaches are tedious and less desirable than having a logs exporter

[dyladan]

We actually discussed this in the last SIG meeting and as far as I remember someone was interested in starting work on this. There were a couple people interested in the topic. Logs work hasn't really started yet in JS since the maintainers are already stretched a bit thin and trying to focus on getting the metrics SDK ready for GA.

[legendecas]

I find that the logs specification is not yet feature-freeze, so it might be worth waiting for the spec to be ready?

[dyladan]

I think the data model is stable and has been for a while. Creating at least the exporter and a basic experimental sdk shouldn't be too much work and is unlikely to be broken by future spec. I don't have time to work on this, but I wouldn't block anyone who wants to work on this. I think @martinkuba mentioned in a SIG meeting he may have some interest?

[martinkuba]

That is correct - I am interested in working on this along with a prototype for the Events API.

[RichiCoder1]

I'm not sure how best I can help, but I'm also interested in this! Been using the Logs Exporter in .NET projects to nice success.

[RichiCoder1]

I did a (very naive) implementation of this for Cloudflare Workers, and am game to take this on or assist too!

Reading https://github.com/open-telemetry/opentelemetry-specification/issues/2506 it sounds like the "API" is currently undefined except the types defined in the SDK spec. Would an @opentelemetry/api-logs just contain the base types for now? With Noop (ConsoleLog?) Emitter.

Following on, would a good MVP be @opentelemetry/sdk-logs-base and a @opentelemetry/exporter-logs-otlp-* set of exporters?

[vmarchaud]

Would an @opentelemetry/api-logs just contain the base types for now? With Noop (ConsoleLog?) Emitter.

I think just having a SDK is the right way, at least thats how java sig went: https://github.com/open-telemetry/opentelemetry-java/tree/main/sdk/logs/src/main/java/io/opentelemetry/sdk/logs

[vmarchaud]

Actually it seems that the API is on its way: https://github.com/open-telemetry/opentelemetry-specification/pull/2676

[RichiCoder1]

Actually it seems that the API is on its way: open-telemetry/opentelemetry-specification#2676

Yup! I figure I can start working from the above assumption and circle back around the API once it's officially landed in the spec. That at least allows instrumenting loggers (and potentially console).

Edit: though the API is looking fairly simple...

[martinkuba]

I put this aside while the API was being discussed, and I have opened a draft implementation of the API.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue was closed because it has been stale for 14 days with no activity.

[martinkuba]

@kushal-ti The Log SDK is now implemented as well as three log exporters https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/exporter-logs-otlp-grpc https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/exporter-logs-otlp-http https://github.com/open-telemetry/opentelemetry-js/tree/main/experimental/packages/exporter-logs-otlp-proto

[pichlermarc]

The initial ask of the issue that fits the scope of this repository is done (exporting Logs via OTLP), so I'm closing this issue. :slightly_smiling_face:

Integrations into common loggers like winston that would allow exporting the logs into OTLP format, similar to traces exporter and metrics exporter, we could have a logs exported

A request for a winston integration would be very fitting for the open-telemetry/opentelemetry-js-contrib repository, which is home to instrumentations that take care of this type of thing. If this is something of interest, please open an issue there. :slightly_smiling_face: