AutoConfiguredOpenTelemetrySdk, when configured with ResourceProviders loses logs on startup

[mikebell90]

Describe the bug A clear and concise description of what the bug is.

If

1. I am using AutoConfiguredOpenTelemetrySdk instead of OpenTelemetrySdk
2. I have opentelemetry-resources in my classpath
3. I am using logback appender.

Then I lose the logs emitted before the install command is called.

If I

1. either use OpenTelemetrySdk or don't have opentelemetry-resources in my classpath

I do not have this issue.

Also if I set a debug endpoint I can see the queued items in the "pre-install" in the logback correctly drained and if I slowly step through the "emit" loop, most of them appear. This implies some sort of race.

Steps to reproduce If possible, provide a recipe for reproducing the error.

1. docker run -p 3000:3000 -p 4317:4317 -p 4318:4318 --rm -ti grafana/otel-lgtm
2. System property pointing to logback file: -Dlogging.config=/Users/mbell/git/service-otj-mvc-demo/src/main/resources/logback-local.xml
3. Env Vars: OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317;OTEL_EXPORTER_OTLP_PROTOCOL=grpc

What did you expect to see? A clear and concise description of what you expected to see. Several hundred log lines that had been queued

What did you see instead? A clear and concise description of what you saw instead. A couple of log lines

What version and what artifacts are you using? Artifacts: (e.g., opentelemetry-api, opentelemetry-sdk, which exporters, etc) Version: (e.g., v0.4.0, 1eb551b, etc) How did you reference these artifacts? (excerpt from your build.gradle, pom.xml, etc)

<opentelemetry-instrumentation-bom.version>2.19.0-alpha</opentelemetry-instrumentation-bom.version>
<dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>io.opentelemetry.instrumentation</groupId>
                <artifactId>opentelemetry-instrumentation-bom-alpha</artifactId>
                <version>${opentelemetry-instrumentation-bom.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>

Environment Compiler: Correttto 17 OS:mac

Additional context Add any other context about the problem here.

Spring 3.5.5. I'm using "Autoconfiguration", but not instrumentation nor Spring Boot AutoConfiguration/starter

If I add -Dotel.java.enabled.resource.providers=io.opentelemetry.sdk.autoconfigure.EnvironmentResourceProvider

which from the docs implies to me this is the ONLY configured ResourceProvider. Things work

this implies to me a faulty resource provider, but I don't know how to find it

logback-local.xml

WireOpenTelemetry.java

DemoServerMain.java

[mikebell90]

I may have found it!

I iteratively built -Dotel.java.enabled.resource.providers and found that io.opentelemetry.instrumentation.resources.ProcessResourceProvideris the culprit

Setting -Dotel.java.disabled.resource.providers=io.opentelemetry.instrumentation.resources.ProcessResourceProvider works around.

Honestly ProcessResourceProvider kind of "sucks" anyway because it builds a gigantic arg line. I wonder if that is behind the issue.

Most of the rest appears benign maybe

[mikebell90]

BTW confirmed switching to http/protobuf made no difference. I suspected it MIGHT because of how large process arg line is (includes all classpath jars). It didn't

[jkwatson]

yeah, the command line resource attribute is very often problematic. I wonder if this is something we should consider removing by default, @jack-berg @trask

[trask]

related discussions: https://github.com/open-telemetry/opentelemetry-java-instrumentation/issues/10151