C++/python interoperability

[nerdvegas]

trafficstars

My (quite large) employer requires distributed tracing in mixed cpp/python environments. From what I've read though, it appears that opentelemetry-python and opentelemetry (the cpp project) are completely separate implementations, correct?

This existing ticket describes the problem we're facing: https://github.com/open-telemetry/community/discussions/734

Do you have any thoughts on how we could most easily implement interoperability between the cpp and python projects? I am interested in interoperability specifically, rather than, say, improving performance by binding the entirety of opentelemetry to python as a separate project. Ideally there would be some way to bind only what is minimally necessary to correctly share state between the two implementations.

Is your feature request related to a problem? Lack of ability to trace from cpp code through to python and vice versa.

Describe the solution you'd like Minimally required py bindings to the cpp project that are compatible with the existing opentelemetry-python project.

Describe alternatives you've considered There don't appear to be any.

[aabmass]

Thanks for digging up that discussion.

From what I've read though, it appears that opentelemetry-python and opentelemetry (the cpp project) are completely separate implementations, correct?

That is correct. I've thought about this a little bit but came to the same conclusions as the linked discussion. Making the same context modifications in both C++ and Python could give minimal interoperability. It would behave similarly to making a propagated call to another process/machine, where any child spans are handled in the process's respective SDKs. You would not be able to share a single span between the two or share export pipelines, samplers, processors etc. IMO, these limitations are not that bad since OTel provides ParentBased sampler to respect the parent span's sampling decision, and the same exporters have been implemented for many languages.

I imagine there is more work to be done on the C++ side, because other users may ask for Ruby/NodeJS/etc. integrations as well.

cc @codeboten and @lzchen who I remember sharing some thoughts on creating a minimal alternative Python SDK that wraps the C++ SDK.

[lalitb]

You would not be able to share a single span between the two or share export pipelines, samplers, processors etc.

I agree it wouldn't be easy to share the export pipelines between different languages. But would that further mean there would be two different exporter pipelines (each for C++ and Python) configured to run in parallel within a single process - to allow exporting spans directly from the C++ component?

I imagine there is more work to be done on the C++ side, because other users may ask for Ruby/NodeJS/etc. integrations as well.

There were discussions to write C API wrapper over C++ SDK which would enable configuring SDK from other languages in ABI compliant way, but this could never be materialized for the shortage of resources.

cc @codeboten and @lzchen who I remember sharing some thoughts on creating a minimal alternative Python SDK that wraps the C++ SDK.

Would be interesting to hear more about that :)

[sirzooro]

Recently I looked how this could be done for NodeJS. At a first glance it seems that it would be enough to create thin NodeJS wrappers for TracerProvider, Tracer, Span, SpanContext and Scope - this should allow to create spans on JS side. I also would have to add custom function to get wrapped TracerProvider from C++ side and set it as a provider on JS side, so both C++ and JS would use the same export pipeline.

I assumed that other pieces could be reused as-is on JS side. If this is not true, more work would be needed. I am not sure about context propagation, it may need extra wrapped classes.