[Logs] Make Logging pipeline extensible to allow plugging in a custom logger provider

[dpk83]

Feature Request

Make Logging pipeline extensible to allow plugging in custom logger provider.

Is your feature request related to a problem?

Currently LogRecord constructor is internal and not exposed which limits the ability for consumers to plug in custom logger provider. This is very restrictive and doesn't go with the OpenTelemetry's open architecture where it allows extensibility points throughout the pipeline e.g. you can plug in your exporter, your processor etc. We need the ability to plug in our custom logger provider so we can introduce additional functionality needed for our customers. One such example is adding a framework that lets customers enrich the logs or doing some custom transformation inside logger etc.

Describe the solution you'd like:

We would like the ability to plug in custom logger. There are 2 ways you can do that

• Expose LogRecord object as public and any public properties on the LogRecord objects to be mutable. This allows a custom logger to create the LogRecord object and then pass it to the OpenTelemetry pipeline.
• Change LogRecord to be an interface, Custom loggers can create their own log record object implementing the interface and that's what is used in OpenTelemetry pipeline and the exporters.

Describe alternatives you've considered.

• We currently are using reflection to create LogRecord object which will break if LogRecord constructor is changed.
• Another option is to fork the code and make the changes in the logger to turn into our custom logger.

We would definitely want to avoid forking the codebase.

Additional Context

[cijothomas]

Once https://github.com/open-telemetry/opentelemetry-dotnet/pull/2864 is merged, it'll allow a good amount of customization. (It wont be part of the coming 1.2 stable).

There is a fairly involved refactoring done to fix some core issues with the way OpenTelemetryLoggerProvider "buffers" state, in this PR https://github.com/open-telemetry/opentelemetry-dotnet/pull/3056 as well. We should wait for this to settle 1st before any changes to logging, as this is a significant improvement, and has impact on Exporters like OTLP as well.

[Falco20019]

I was also looking into creating a Serilog sink for OpenTelemetry and failed also due to the LogRecord not having a public ctor. So I'm also looking forward to this :) Right now, I'm doing a double looping by creating a sink using ILogger as intermediate, which sadly has some downsides like the timestamp not being correct.

[RichiCoder1]

@Falco20019 howdy! Did you end up doing further work on a Serilog OTLP Sink? It looks like that while LogRecord's ctor isn't public, there's at least now a way to make them (LogRecordPool)

[Falco20019]

No, I didn't follow up since I had to drop support for OTEL all together since it was not supporting Xamarin.Android. I might have another look the next months since it seems to work better with net6.0-android and becomes more interesting through that. The current plan is to look into OTLP for our release in January.