Fixes #30712

Jan 17 '24 09:01 loicmathieu

/cc @brunobat (opentelemetry), @radcortez (opentelemetry)

Jan 17 '24 09:01 quarkus-bot[bot]

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building fee7c478bfc594599719357cf9cad559d94016ed

Status	Name	Step	Failures	Logs	Raw logs	Build scan

Jan 17 '24 09:01 quarkus-bot[bot]

Thanks for the PR @loicmathieu. I have a concern in relation to configuration. The way it's organised right now, you will not be able to activate/deactivate OTel Logging from the main quarkus-opentelemetry extension. I wonder if this code could be placed under quarkus-opentelemetry and integrated with all the other OTel configs?

Jan 17 '24 10:01 brunobat

@brunobat we can move all configuration properties to the main otel extension and consume this property inside the logs extension. Or we can do the opposite, move what is needed for logs here (by the way, some config has been removed so my extension didn't work anymore) and only check that otel is enabled globally or not.

We need to settle this as we would do the same for other otel related extension when created (for ex for metrics).

Jan 17 '24 11:01 loicmathieu

@loicmathieu Metrics will be a very different beast, as we don't plan to use its API inside Quarkus automatic instrumentation. We need a bridge from Micrometer -> OTel and I'm currently working on that. In the long run we might need an Observability extension to integrate all these different functionalities.

One thing is certain, there will be a single OTel SDK for all signals and all output will use the OTLP protocol from that SDK.

This means all OTel related configurations must line in quarkus-opentelemetry because of the transparent integration with the OTel SDK Autoconfiguration and its standard properties. However, if some functionality (logs) can only be used if an additional extension is available, that can be weird. Ideally, the functionality itself should live in the OTel extension and the code should be activated if the right configs are On. @geoand Do you have an opinion on this?

Jan 17 '24 12:01 brunobat

OK, there is not a log of code to support logs so it could easily be merged into the existing otel extension.

Jan 17 '24 12:01 loicmathieu

@geoand Do you have an opinion on this?

Not really :)

Jan 17 '24 12:01 geoand

@loicmathieu do you mind if I do some commits on the PR? I have some ideas about how to integrate and test it... Probably just on next week.

Feb 13 '24 16:02 brunobat

@brunobat of course you can add commits to my PR, be my guest ;)

For now, this didn't work as the Log tracer uses OkHttp which is no longer in the classpath (it was in the classpath when I made the first implementation) so there is still works to do.

Feb 14 '24 09:02 loicmathieu

Following the recent Keycloak blog post on the deprecation of GELF log outputs, it's evident that we need to transition to an alternative logging format, such as OTEL. Ideally, we aim to avoid reverting to manual log file analysis, which would be a significant step backward.

GELF log handler has been deprecated With sunsetting of the underlying library providing integration with GELF, Keycloak will no longer support the GELF log handler out-of-the-box. This feature will be removed in a future release. If you require an external log management, consider using file log parsing.

Mar 12 '24 16:03 crumohr

@loicmathieu is there any way that we can offer support in getting this PR merged?

May 02 '24 07:05 crumohr

Once merged https://github.com/quarkusio/quarkus/pull/39032, we should have a vert.x exporter that we can use with this PR.

May 06 '24 08:05 brunobat

@loicmathieu The metrics commit got merged last night, so this would now be unblocked as far as I understand. Can you rebase your work?

Jul 09 '24 06:07 jhenkehd

@loicmathieu now we can work on this PR. Do you think you could use these senders, like the other signals?

Jul 10 '24 18:07 brunobat

@brunobat I'll look at this soon.

Jul 11 '24 07:07 loicmathieu

Hey @loicmathieu,

is there an update on this? Is it realistic to get this in one of the upcoming Quarkus versions or should we look into other ways to get logs via OTEL?

Thanks for your work on this!

Jul 22 '24 13:07 andreas-eberle

Hi, I plan to find some time to work on it soon, but as the next Quarkus (3.13) is already forked from mainline since one week it would not be available before 3.14 at least.

Jul 22 '24 13:07 loicmathieu

Since the next LTS release will be based on the 3.14 feature set, it would be quite important to get it merged in time for 3.14. I think it would be great, if we have complete OTel support in the next LTS.

Do you think that would be realistic? Do you need help to get it ready for the 3.14 merge window?

Jul 22 '24 13:07 janhenke

Feature freeze for 3.14 LTS will be on August 14th, is there any way we can help with getting this PR ready?

Aug 08 '24 11:08 crumohr

@brunobat I implemented the log exporter inspired by the metric one but the unit test still complain about missing OkHttp.

2024-08-09 18:18:31,290 INFO  [io.ope.api.GlobalOpenTelemetry] (Quarkus Main Thread) AutoConfiguredOpenTelemetrySdk found on classpath but automatic configuration is disabled. To enable, run your JVM with -Dotel.java.global-autoconfigure.enabled=true

2024-08-09 18:18:31,415 ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:119)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:116)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.NoClassDefFoundError: okhttp3/RequestBody

Autoconfiguration seems to be off, so I should have missed something somewhere.

Aug 09 '24 16:08 loicmathieu

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

Status for workflow `Quarkus Documentation CI`

This is the status report for running Quarkus Documentation CI on commit abc8100f4baceacb91abb283e78ecf9662f5d240.

Failing Jobs

:warning: Artifacts of the workflow run were not available thus the report misses some details.

Status	Name	Step	Failures	Logs	Raw logs	Build scan
✖	Documentation Build	`Build`	:warning: Check →	Logs	Raw logs	:construction:

Aug 09 '24 16:08 quarkus-bot[bot]

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

Status for workflow `Quarkus Documentation CI`

This is the status report for running Quarkus Documentation CI on commit 9de9b635725d01a3c23c5ab2e7fcdf5bedffac38.

Failing Jobs

:warning: Artifacts of the workflow run were not available thus the report misses some details.

Status	Name	Step	Failures	Logs	Raw logs	Build scan
✖	Documentation Build	`Build`	:warning: Check →	Logs	Raw logs	:construction:

Aug 12 '24 12:08 quarkus-bot[bot]

@brunobat I had configuration issues on the integration test that's why it didn't worked. Now, the application is started correctly but nothings is sent to the collector. I need to investigate a little more.

Aug 12 '24 12:08 loicmathieu

@gsmet @geoand @brunobat this PR is almost ready, it misses documentation. As 3.14 will be cut off soon and people are eager to have it in the next LTS can this be reviewed and merged as is?

I'll provide followup PRs for doc and improvements.

@brunobat the implementation is not ideal as I need the OpenTelemetry at RUNTIME_INIT to build the of handler but if I do this it crashed the application so I had to get it for each log record, see https://github.com/quarkusio/quarkus/pull/38239/files#diff-fb20629bdd702f46a2bfd6f8266227aac4f9c2c72daffc4254943105d9641020R17 not sure it's ideal. Is there a way to get an OpenTelemetry object at RUNTIME_INIT, it seems to be initialized very late.

Aug 13 '24 11:08 loicmathieu

Thanks for this!

I'm on PTO, so I won't be able to review this in time

Aug 13 '24 12:08 geoand

:waning_crescent_moon: This workflow status is outdated as a new workflow run has been triggered.

Status for workflow `Quarkus CI`

This is the status report for running Quarkus CI on commit 08fd96fb476b93a4e362b347eb9a95e2b6af0076.

[!WARNING] Unable to include the stracktraces as the report was too long. See annotations below for the details.

[!WARNING] Unable to include the failure links as the report was too long. See annotations below for the details.

Failing Jobs

Status	Name	Step	Failures	Logs	Raw logs	Build scan
✖	JVM Tests - JDK 17	`Build`	Failures	Logs	Raw logs	:mag:
✖	JVM Tests - JDK 21	`Build`	Failures	Logs	Raw logs	:mag:
✖	JVM Tests - JDK 17 Windows	`Build`	Failures	Logs	Raw logs	:mag:
✖	Native Tests - AWT, ImageIO and Java2D, Packaging .so files	`Set up job`	:warning: Check →	Logs	Raw logs	:construction:
✖	Native Tests - DevTools Integration Tests	`Set up job`	:warning: Check →	Logs	Raw logs	:construction:
✖	Native Tests - HTTP	`Build`	Failures	Logs	Raw logs	:mag:
✖	Native Tests - Misc4	`Build`	Failures	Logs	Raw logs	:mag:

Full information is available in the Build summary check run. You can consult the Develocity build scans.

Failures

:gear: JVM Tests - JDK 17 #

- Failing: extensions/opentelemetry/deployment integration-tests/grpc-cli 
! Skipped: extensions/liquibase-mongodb/deployment extensions/micrometer-registry-prometheus/deployment extensions/micrometer/deployment and 50 more

:package: extensions/opentelemetry/deployment

✖ io.quarkus.opentelemetry.deployment.InvalidAttributesConfigTest. - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryContinuousTestingTest.testContinuousTesting - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryDestroyerTest.getShutdownWaitTime - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryDevModeTest.testDevMode - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryDevServicesDatasourcesTest.devDatasource - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryMDCTest. - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryResourceTest. - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryServiceNameAppNameTest. - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryServiceNameCombinedResourceWinsTest. - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryServiceNameCombinedServiceWinsTest. - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryServiceNameNoResourceAttrsTest. - History

✖ io.quarkus.opentelemetry.deployment.OpenTelemetryServiceNameResourceAttrTest. - History

✖ io.quarkus.opentelemetry.deployment.exporter.otlp.OtlpExporterBadEndpointTest. - History

✖ io.quarkus.opentelemetry.deployment.exporter.otlp.OtlpExporterConfigTest. - History

✖ io.quarkus.opentelemetry.deployment.exporter.otlp.OtlpExporterOverrideConfigTest. - History

✖ io.quarkus.opentelemetry.deployment.exporter.otlp.OtlpTraceExporterDisabledTest. - History

✖ io.quarkus.opentelemetry.deployment.instrumentation.GraphQLOpenTelemetryTest. - History

✖ io.quarkus.opentelemetry.deployment.instrumentation.GrpcOpenInstrumentationDisabledTest. - History

✖ io.quarkus.opentelemetry.deployment.instrumentation.GrpcOpenTelemetryTest. - History

✖ io.quarkus.opentelemetry.deployment.instrumentation.RestClientOpenTelemetryTest. - History

✖ io.quarkus.opentelemetry.deployment.instrumentation.VertxClientOpenTelemetryTest. - History

✖ io.quarkus.opentelemetry.deployment.instrumentation.VertxEventBusInstrumentationDisabledTest. - History

✖ io.quarkus.opentelemetry.deployment.instrumentation.VertxHttpInstrumentationDisabledTest. - History

✖ io.quarkus.opentelemetry.deployment.instrumentation.VertxOpenTelemetryForwardedTest. - History

✖ io.quarkus.opentelemetry.deployment.instrumentation.VertxOpenTelemetryTest. - History

✖ io.quarkus.opentelemetry.deployment.instrumentation.VertxOpenTelemetryXForwardedTest. - History

✖ io.quarkus.opentelemetry.deployment.interceptor.AddingSpanAttributesInterceptorTest. - History

✖ io.quarkus.opentelemetry.deployment.interceptor.WithSpanInterceptorTest. - History

✖ io.quarkus.opentelemetry.deployment.metrics.GaugeCdiTest. - History

✖ io.quarkus.opentelemetry.deployment.metrics.MetricsTest. - History

✖ io.quarkus.opentelemetry.deployment.propagation.OpenTelemetryMpContextPropagationTest. - History

✖ io.quarkus.opentelemetry.deployment.propagation.OpenTelemetryTextMapPropagatorCustomizerTest. - History

✖ io.quarkus.opentelemetry.deployment.traces.NonAppEndpointsDisabledTest. - History

✖ io.quarkus.opentelemetry.deployment.traces.NonAppEndpointsDisabledWithRootPathTest. - History

✖ io.quarkus.opentelemetry.deployment.traces.NonAppEndpointsEnabledLegacyConfigurationTest. - History

✖ io.quarkus.opentelemetry.deployment.traces.NonAppEndpointsEnabledTest. - History

✖ io.quarkus.opentelemetry.deployment.traces.OpenTelemetryCustomSamplerBeanTest. - History

✖ io.quarkus.opentelemetry.deployment.traces.OpenTelemetryHttpCDITest. - History

✖ io.quarkus.opentelemetry.deployment.traces.OpenTelemetryIdGeneratorTest. - History

✖ io.quarkus.opentelemetry.deployment.traces.OpenTelemetryReactiveRoutesTest. - History

✖ io.quarkus.opentelemetry.deployment.traces.OpenTelemetrySamplerBeanTest. - History

✖ io.quarkus.opentelemetry.deployment.traces.OpenTelemetrySamplerConfigTest. - History

✖ io.quarkus.opentelemetry.deployment.traces.OpenTelemetrySpanSecurityEventsTest. - History

:package: integration-tests/grpc-cli

✖ io.quarkus.grpc.cli.GrpcCliTest.testCommand - History

:gear: JVM Tests - JDK 21 #

- Failing: extensions/opentelemetry/deployment integration-tests/grpc-cli 
! Skipped: extensions/liquibase-mongodb/deployment extensions/micrometer-registry-prometheus/deployment extensions/micrometer/deployment and 50 more