opentelemetry-go icon indicating copy to clipboard operation
opentelemetry-go copied to clipboard

exporters: add otlplogfile exporter

Open thomasgouveia opened this issue 1 year ago • 2 comments

This PR introduces an experimental otlplogfile exporter based on the following specification: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/file-exporter.md.

For the implementation, I decided to use a buffered file writer based on the existing fileexporter available in the opentelemetry-collector to reduce I/O and improve performance. Below is benchmark result:

goos: darwin
goarch: arm64
pkg: go.opentelemetry.io/otel/exporters/otlp/otlplog/otlplogfile/internal/writer
cpu: Apple M1 Pro
BenchmarkBufferedWriter
BenchmarkBufferedWriter/raw-file_10485760_bytes
BenchmarkBufferedWriter/raw-file_10485760_bytes-8         	36347986	        31.49 ns/op	      48 B/op	       1 allocs/op
BenchmarkBufferedWriter/buffered-file_10485760_bytes
BenchmarkBufferedWriter/buffered-file_10485760_bytes-8    	442928779	         2.704 ns/op	       0 B/op	       0 allocs/op
BenchmarkBufferedWriter/raw-file_104857600_bytes
BenchmarkBufferedWriter/raw-file_104857600_bytes-8        	34564147	        31.50 ns/op	      48 B/op	       1 allocs/op
BenchmarkBufferedWriter/buffered-file_104857600_bytes
BenchmarkBufferedWriter/buffered-file_104857600_bytes-8   	439031508	         2.703 ns/op	       0 B/op	       0 allocs/op
BenchmarkBufferedWriter/buffered-file_1073741824_bytes
BenchmarkBufferedWriter/buffered-file_1073741824_bytes-8  	400000000	         2.681 ns/op	       0 B/op	       0 allocs/op
BenchmarkBufferedWriter/raw-file_1073741824_bytes
BenchmarkBufferedWriter/raw-file_1073741824_bytes-8       	31952744	        35.17 ns/op	      48 B/op	       1 allocs/op
BenchmarkBufferedWriter/raw-file_10737418240_bytes
BenchmarkBufferedWriter/raw-file_10737418240_bytes-8      	       1	5706859250 ns/op	      32 B/op	       2 allocs/op
BenchmarkBufferedWriter/buffered-file_10737418240_bytes
BenchmarkBufferedWriter/buffered-file_10737418240_bytes-8 	       1	5981516417 ns/op	       0 B/op	       0 allocs/op
PASS

To serialize the records to the OTLP JSON format, I used protojson.

Closes #5408

thomasgouveia avatar Aug 27 '24 12:08 thomasgouveia

I do not think we should accept this change. It adds an exporter that has not been stabilized at the specification level. This addition will add churn to our stabilization efforts of the log signal.

I do not see why this cannot live in a fork as a PoC for the specification, or as an independent repository with its own maintainers outside of OTel. Adding this to the OTel development teams maintenance burden does not seem warranted at this time.

MrAlias avatar Sep 10 '24 18:09 MrAlias

From the SIG meeting:

  • This is currently in development, and we aren't required to implement it
  • This is a large amount of code, and will likely be a burden on maintainers as it evolves
  • We would prefer if this lived in its own repository until it is stabilized and required for us to implement

dashpole avatar Sep 12 '24 17:09 dashpole

Changing to draft as there is not work here currently.

pellared avatar Nov 08 '24 08:11 pellared