census-instrumentation
Add ability to create several trace components within one JVM
Currently, there is no ability to create several same TraceComponent's within one JVM. The only way is to use singleton located in io.opencensus.trace.Tracing class.
If I try to create TraceComponent manually using existing code io.opencensus.trace.Tracing#loadTraceComponent it fails on second component creation because of failed metrics registration:
java.util.ServiceConfigurationError: Provider io.opencensus.impl.trace.TraceComponentImpl could not be instantiated.
at io.opencensus.internal.Provider.createInstance(Provider.java:46)
at org.apache.ignite.opencensus.spi.tracing.OpenCensusTracingSpi.loadTraceComponent(OpenCensusTracingSpi.java:97)
at org.apache.ignite.opencensus.spi.tracing.OpenCensusTracingSpi.spiStart(OpenCensusTracingSpi.java:74)
at org.apache.ignite.internal.processors.tracing.TracingProcessor.start(TracingProcessor.java:33)
at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1849)
at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1114)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:1988)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1688)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1111)
at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:609)
at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:970)
at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:911)
at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:899)
at org.apache.ignite.testframework.junits.GridAbstractTest.startGrid(GridAbstractTest.java:865)
at org.apache.ignite.testframework.junits.GridAbstractTest.startGrids(GridAbstractTest.java:754)
at org.apache.ignite.opencensus.spi.tracing.OpenCensusTracingSpiTest.before(OpenCensusTracingSpiTest.java:31)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.apache.ignite.testframework.junits.GridAbstractTest$6.run(GridAbstractTest.java:2069)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at io.opencensus.internal.Provider.createInstance(Provider.java:43)
... 26 more
Caused by: java.lang.IllegalArgumentException: A different time series with the same labels already exists.
at io.opencensus.implcore.metrics.DerivedLongCumulativeImpl.createTimeSeries(DerivedLongCumulativeImpl.java:96)
at io.opencensus.implcore.trace.export.SpanExporterImpl.<init>(SpanExporterImpl.java:125)
at io.opencensus.implcore.trace.export.SpanExporterImpl.create(SpanExporterImpl.java:89)
at io.opencensus.implcore.trace.export.ExportComponentImpl.<init>(ExportComponentImpl.java:83)
at io.opencensus.implcore.trace.export.ExportComponentImpl.createWithInProcessStores(ExportComponentImpl.java:63)
at io.opencensus.implcore.trace.TraceComponentImplBase.<init>(TraceComponentImplBase.java:60)
at io.opencensus.impl.trace.TraceComponentImpl.<init>(TraceComponentImpl.java:40)
... 31 more
We're trying to integrate opencensus tracing module to Apache Ignite project and we have a case with starting several our nodes within one JVM. Each of the nodes has own independent set of components and tracing component couldn't be created separately because of the issue above. There is a workaround with pre-creating TraceComponent once and sharing it between several nodes, but negative side-effects because of that action can't be predicted yet. I think it would be more convenient to give the possibility of separation of several trace components within one JVM, WDYT?
Hi @Jokser, why do you want each node to have a new trace component? The reason we limit one trace component is that we want to keep some global state. For example all sampled spans of one JVM should go into one single exporter thread, which then get dispatched to all registered handlers. While if we had multiple instances of trace components, it becomes difficult to manage the registry, synchronization, duplication, etc.
There is a workaround with pre-creating TraceComponent once and sharing it between several nodes, but negative side-effects because of that action can't be predicted yet.
This should be fairly easy since each node can get the global components with Tracing.getTracer/Tracing.getExportComponent etc.
Hi @songy23, thank you for your response! The main problem is how to distinguish traces from different services located in the same JVM. I mean service in terms of Zipkin exporter configuration. As I see the exporter has "service" field which represents a logical separation between some business and internal processes. Let's suppose the situation if you have some web-service that does some business logic and embedded database (Ignite) for storing/querying a data to it. Physically they share the same JVM but logically it's two different service - business and db. At the moment I can use tags to separate spans come from business service and db but both of them will relate to the same service. If separation of tracing components is difficult task at the moment Is there any way to filter out some spans that some of them will be exported by one handler and some of them by another with different service names?
If separation of tracing components is difficult task at the moment Is there any way to filter out some spans that some of them will be exported by one handler and some of them by another with different service names?
I see your point, thanks for the explanation. Unfortunately it's difficult to implement the multi-instance tracing components, and we currently don't have the ability in OpenCensus to filter certain spans for particular exporting handlers. However in OpenTelemetry (the successor of OpenCensus), spans can have a resource which holds information like service names.
Also please feel free to file an issue to https://github.com/open-telemetry/opentelemetry-specification about adding the ability on filtering spans when exporting.
@songy23 Thank you for mentioning OpenTelemetry. Will file the issue there regarding the problem.