open-telemetry
Able to see traces with Java, but no trace with Scala (same code)
Describe the bug A clear and concise description of what the bug is.
=> I am able to send traces with below Java code, I can even see the traces in our tracing back-ends.
However, what seems to be the same code (can be discussed) in Scala will yield no trace at all.
Both are using the same OpenTelemetry API, SDK and exporter OTLP
Steps to reproduce If possible, provide a recipe for reproducing the error.
Here is the java code. I hope it is as minimal and as descriptive as possible.
import io.opentelemetry.api.OpenTelemetry;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.trace.*;
import io.opentelemetry.context.Context;
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter;
import io.opentelemetry.sdk.OpenTelemetrySdk;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.sdk.trace.export.SimpleSpanProcessor;
public class AAA {
public String call(String traceParent) throws Exception {
//String traceParent = "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"; //for testing
OtlpGrpcSpanExporter spanExporter = OtlpGrpcSpanExporter.builder()
.setEndpoint("https://tempo-prod-04-prod-us-east-0.grafana.net:443")
.addHeader("Authorization", "Basic ODEwNzcxOmdsY19leUp2SWpvaU1[...]luQnliMlF0ZFhNdFpXRnpkQzB3SW4xOQ==")
.build();
Resource myResource = Resource.create(Attributes.of(AttributeKey.stringKey("service.name"), "javaservice"));
SdkTracerProvider tracerProvider = SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)).build();
OpenTelemetry openTelemetry = OpenTelemetrySdk.builder().setTracerProvider(tracerProvider).build();
String[] traceSplit = traceParent.split("-");
SpanContext remoteContext = SpanContext.createFromRemoteParent(traceSplit[1], traceSplit[2], TraceFlags.getSampled(), TraceState.getDefault());
Tracer tracer = openTelemetry.getTracer("from-java");
SpanBuilder sb= tracer.spanBuilder("java-name").setSpanKind(SpanKind.CONSUMER);
sb.setParent(Context.current().with(Span.wrap(remoteContext)));
sb.setAttribute("testattributejava", "testattributejava");
Span serverSpan = sb.startSpan();
try {
Thread.sleep(100); // mock our operation
System.out.println("This is an example operation." + traceSplit[1] + traceSplit[2] + serverSpan + serverSpan.isRecording() + remoteContext);
return serverSpan + " " + remoteContext;
} finally {
serverSpan.end();
}
}
}
And here is the Scala counterpart, not working.
import io.opentelemetry.api.common.Attributes
import io.opentelemetry.api.trace._
import io.opentelemetry.context.Context
import io.opentelemetry.exporter.otlp.trace.OtlpGrpcSpanExporter
import io.opentelemetry.sdk.OpenTelemetrySdk
import io.opentelemetry.sdk.resources.Resource
import io.opentelemetry.sdk.trace.SdkTracerProvider
import io.opentelemetry.sdk.trace.`export`.SimpleSpanProcessor
import org.apache.spark.sql.Row
class Issue {
def call(traceParent: (String, String)): String = {
val spanExporter = OtlpGrpcSpanExporter.builder().setEndpoint("https://tempo-prod-04-prod-us-east-0.grafana.net:443").addHeader("Authorization", "Basic ODEwNzcxOmdsY19leUp2SWpvaU1UQTFOVFV4T1NJc0ltNGlPaUp6ZEdGamF5MDROVGcyTnpJdG[...]npkQzB3SW4xOQ==").build()
val myResource = Resource.create(Attributes.builder().put("service.name", "scalaservice").build())
val tracerProvider = SdkTracerProvider.builder().addSpanProcessor(SimpleSpanProcessor.create(spanExporter)).build()
val openTelemetry = OpenTelemetrySdk.builder().setTracerProvider(tracerProvider).build()
val remoteContext = SpanContext.createFromRemoteParent(traceParent._1, traceParent._2, TraceFlags.getSampled, TraceState.getDefault)
val tracer = openTelemetry.getTracer("from-scala")
val spanBuilder = tracer.spanBuilder("scala-name").setSpanKind(SpanKind.CONSUMER)
spanBuilder.setParent(Context.current().`with`(Span.wrap(remoteContext)))
spanBuilder.setAttribute("testAttributeScala1", "testAttributeScala2")
val mySpan = spanBuilder.startSpan()
try {
Thread.sleep(400) //mock our work
println(traceParent._1 + traceParent._2 + mySpan.toString + mySpan.isRecording + remoteContext)
mySpan.toString
} finally {
mySpan.end()
}
}
What did you expect to see? A clear and concise description of what you expected to see.
=> For both Java and Scala, I was hoping to see the trace in our tracing back ends. (Tested with Tempo, Lightstep, Datadog, Elastic APM)
What did you see instead?
=> We see the trace being sent for the Java code, but not for the Scala counterpart.
What version and what artifacts are you using?
Artifacts: => opentelemetry-api, opentelemetry-sdk, 1.49.0 which exporters => OTLP
Version: => 1.49.0
How did you reference these artifacts? (excerpt from your build.gradle, pom.xml, etc)
=> Here is the part from SBT
"io.opentelemetry" % "opentelemetry-api" % "1.49.0",
"io.opentelemetry" % "opentelemetry-sdk" % "1.49.0",
"io.opentelemetry" % "opentelemetry-exporter-otlp" % "1.49.0",
Environment Compiler: GraalVM 24, but we are not building native image OS: The latest Noble
Additional context
Hello team,
This is my first issue in this repo.
If not anything else, just wanted to say thanks for the work.
Just wanted to reach out with a small issue.
We have a Java Spark Structured Streaming pipeline with kafka, where we would like to trace calls between Kafka producers (from different sources and languages) to our Spark job.
The above Java code is one step of the map reduce from Spark.
It is working. We can see traces in the different tracing backend.
However, a similar Spark Streaming job in Scala, with the similar map function (tracing) will not work.
No matter what, we cannot see the trace in any of the back end systems.
Could you please help on this issue?
Good day!
With this fairly artificial example (building an SDK and then throwing it away immediately), I assume that there's probably some timing issue that works differently in scala than in Java. SDKs are intended to be long-lived, as the threads that execute exports only happen periodically.
I think if you want to figure out why this isn't working in scala, you might need to really dig in and debug what's happening inside the OTel SDK, expecially with the SpanProcessor (you really don't want to use the simple span processor for real-world use-cases, either), and the exporter.
Hello @ryleighmikami
Can you add Thread.sleep(2000) at the end of your code and let us know if you can see the traces?
@jkwatson
I stumbled into this issue, and the reproducible is there.
You are correct, I managed to reproduce, it is working with java, but ot for scala. But If I add Thread.sleep into the scala code, it is working fine.
I think you are correct that there are different clocks.
Furthermore, I tried the simple span processor, the multi, the batch, and all of them yield the same results as OP mentioned.
May I ask you @jkwatson
-
If there are any tips to making the SDK long lived?
-
Most of all, what is the strategy to send the trace? Is the algorithm some kind of back end thread, which accepts some spans, and periodically sends to the back end tracing service? Can we configure this period?
May I ask you @jkwatson
- If there are any tips to making the SDK long lived?
- Most of all, what is the strategy to send the trace? Is the algorithm some kind of back end thread, which accepts some spans, and periodically sends to the back end tracing service? Can we configure this period?
I recommend reading the docs on opentelemetry.io. All of this is documented in a detailed manner there.