open-telemetry
[Donation Proposal]: Kotlin Multiplatform implementation of OpenTelemetry specification
Description
embrace.io wishes to donate a Kotlin implementation of the OTel specification. This has been developed as a Kotlin Multiplatform Library and currently runs on Android, JVM, iOS, and JavaScript targets.
Modes
The project can run in two modes:
- Compatibility mode: where the Kotlin API binds opentelemetry-java by calling an appropriate bridge implementation under the hood.
- Implementation mode: where the Kotlin API runs the Kotlin implementation
Compatibility mode is primarily intended for folks who are already using opentelemetry-java on Android, but who prefer to use a Kotlin API, or as a bridge for users who want to migrate between the two APIs. Implementation mode is intended for everybody else, and particularly for anybody who uses Kotlin Multiplatform in Android/iOS mobile apps.
What is implemented?
The project has working implementations of the Tracing and Logging APIs. Some smaller features such as specific exporters or sampling are currently not implemented, but continue to progress every day.
Platform support
embrace.io has primarily been interested in targeting this library on Android as that’s our immediate business need. However, Kotlin Multiplatform does allow targeting Kotlin code at a variety of platforms & we have confirmed the project can run on a JavaScript & iOS targets although neither have been put into our production systems.
Given that Kotlin Multiplatform is becoming much more common in Android/iOS apps, we’re hopeful that this could be a useful solution for anybody who wants to write telemetry in one place and have consistent conventions & behavior between different platforms.
Benefits to the OpenTelemetry community
Kotlin is the de facto standard language used in modern Android apps & the OpenTelemetry project does not currently have a dedicated Kotlin SDK or API. Whilst Android developers could use opentelemetry-java via Kotlin’s interoperability, a lot of API design choices such as builders and thread-local context storage feel non-idiomatic when called from Kotlin.
Our hope is that developing a more mobile-friendly API will increase the adoption of OpenTelemetry on Android, and attract more engineers to the community who will have a client-side perspective on telemetry.
We also believe this would be a very useful addition for anybody sharing code using Kotlin Multiplatform, as it would allow them to capture telemetry consistently between different platforms.
Reasons for donation
We would like more folks to contribute to the project, and feel that donating to OpenTelemetry is the best way of achieving that. I also feel that diversity of opinion and interests will be helpful. For example, folks outside of embrace.io might have much stronger interests in developing support for different Kotlin Multiplatform targets, such as iOS/JavaScript.
Repository
https://github.com/embrace-io/opentelemetry-kotlin
Existing usage
embrace.io ships an Android SDK that allows developers to capture telemetry in their apps; opentelemetry-kotlin is used internally within this SDK. The library has been tested across millions of user sessions for several customers in this way.
opentelemetry-kotlin can run in two modes: by binding the opentelemetry-java API, or by using a Kotlin implementation of the OTel specification. The first mode has run in production & the second mode will do so in the near future.
Additionally, the API for opentelemetry-kotlin is exposed to consumers of embrace’s SDK, or anybody else who wants to consume the artifact from Maven Central. We don’t have insight into whether any folks have consumed the library in this way.
Maintenance
Currently three engineers from embrace.io (@bidetofevil, @fractalwrench, @priettt) are willing to act as maintainers.
We would be very happy to welcome maintainers and contributors from additional organisations. Our hope is that by opening this issue, individuals or organisations who are interested in maintaining this project will come forward.
Licenses
Apache 2.0
Trademarks
None, although the package namespace is currently io.embrace.opentelemetry.kotlin. We didn’t want to use the io.opentelemetry namespace until this project was officially donated.
Other notes
Longer-term we feel like the opentelemetry-android project should build on top of opentelemetry-kotlin rather than opentelemetry-java, mainly due to the mobile-friendliness issues previously discussed. This is something that we can talk through in more depth with the Android SIG. We recognise that there will probably be many things to discuss & change before that is a feasible option.
Hanson also gave a talk at OTel Community Day that gives some context on the ‘why’ behind this donation: https://www.youtube.com/watch?v=di5nhYvUh6w
Thanks for starting this thread. I'm excited about Kotlin supporting the OTel spec. A couple of questions...
-
How do the current proposers (and potential maintainers) see the community grow around Kotlin in OpenTelemetry? a. How do you see the Kotlin SIG interact with the Java community in OTel - for features and for collaboration? b. Do you see a potential merger with the Android OTel community in the future given the "mobile-friendliness" you refer to in the proposal?
-
How do you plan to grow a more varied group of maintainers from a sustainability pov?
How do the current proposers (and potential maintainers) see the community grow around Kotlin in OpenTelemetry?
We hope to attract more contributors to this project and think that folks using Kotlin Multiplatform will want to help build out the non JVM/Android targets (i.e. iOS, JavaScript). I'd like to see folks adopt this project on Android/JVM environments & write instrumentation against its API. We hope this will bring folks into the OTel community with a take on things from a client rather than backend perspective, which would increase diversity of opinions.
How do you see the Kotlin SIG interact with the Java community in OTel - for features and for collaboration?
Due to the majority of folks using Kotlin on the JVM/Android I'd anticipate there would be close collaboration & overlap between the two. The Kotlin project is designed with a compatibility layer that allows calling opentelemetry-java's API and vice versa. Having said that, Kotlin these days is a language that just happens to run on the JVM/Android, so there will be cases where it makes sense to forge a different path.
Do you see a potential merger with the Android OTel community in the future given the "mobile-friendliness" you refer to in the proposal?
I think there will be substantial overlap, similar to how there is overlap between opentelemetry-java and opentelemetry-android now. I would certainly love to get to a point where opentelemetry-android feels comfortable building on the Kotlin API exposed in opentelemetry-kotlin. However, I do think it makes sense to retain some separation due to the number of different platforms that Kotlin can support - particularly on iOS & the backend its popularity is growing.
How do you plan to grow a more varied group of maintainers from a sustainability pov? This is a bit of a chicken/egg situation. We think the demand is there for a Kotlin implementation of OpenTelemetry, but also finding suitable maintainers who are willing to contribute to the community might be hard while the project is 'unofficial'. The first step we've taken is to open this donation and share it in various communities to alert folks who might be interested. We also plan on reaching out directly to individuals who we think would be a suitable fit and asking whether they'd be happy to contribute.
To folks who are reading this issue and potentially interested in contributing to the project - please do get in touch or leave a comment!
Thanks for the responses @fractalwrench, it is indeed great to see this proposal taking shape. Building on what Alolita mentioned (and you've cleared stated in the proposal) around building a community around opentelemetry-kotlin, and as discussed during the GC meeting on Sep 17, we'd like to see engineers from more organisations (ideally at least two more) committed to supporting this project before we proceed with the donation process.
For anyone reading that'd like to form part of this group and build this community, please leave a comment.
In terms of extending the reach of this issue, thanks for sharing in different SIG meetings and other places. As an extension to this, may I propose that you write a short blogpost on the OpenTelemetry website, calling for contributors to start a Kotlin implementation of OpenTelemetry, mentioning the donation proposal and directing people to comment on this issue? I'm happy to sponsor/review this blogpost.
@danielgblanco a blogpost sounds like a great idea & I'd be happy to work on it. Are there any guidelines/style guides for how to contribute blog posts on the OpenTelemetry website?
Would it also be possible to clarify what 'committed to supporting this project' would mean for an individual? This would help us set the necessary expectations when we have conversations with people we believe might be interested.
@danielgblanco a blogpost sounds like a great idea & I'd be happy to work on it. Are there any guidelines/style guides for how to contribute blog posts on the OpenTelemetry website?
https://opentelemetry.io/docs/contributing/blog/ -- take a look here. Note that we are going to add a few words about donations to that page eventually, but nothing for you to worry about right now.
Thanks. I've created an issue for a blog post as suggested in the guide and will wait for it to be reviewed: https://github.com/open-telemetry/opentelemetry.io/issues/7809
I support this effort and @fractalwrench and the Embrace crew have been a big help in Android.
I really hope this comment doesn't add any chaos or detract from this effort, that's not my intention! I hadn't heard this mentioned here, so I wanted to point this out: Users who land in the registry today will find ONE item under "Kotlin" as the language, and it's this 3rd party port that claims to be an API and SDK port to Kotlin Multiplatform.
It was added in this PR https://github.com/open-telemetry/opentelemetry.io/pull/1145 over 3 years ago and seems to be from a German company that has a github org with only one repo that contains that project. Weird.
So does anybody know anything about that other project? Have snk/dcxp been active elsewhere in otel?
We did spot this project - it's a 1:1 port of opentelemetry-java and I think it's ultimate goal was to refactor the API to make things more Kotlin friendly. I don't think it currently supports OTLP export or platforms other than JVM. I can attempt to reach out to someone at that company if nobody else has contact with them.
Slight tangent: would it be acceptable for Embrace to open a PR to add opentelemetry-kotlin to that registry, while we wait for the donation proposal proceeds? I don't want to cause confusion but think it could be helpful for folks who are searching for a Kotlin implementation of OpenTelemetry.
I'd be interested in contributing. I have a background in Android development but new to the observability/OTel space.
Slight tangent: would it be acceptable for Embrace to open a PR to add opentelemetry-kotlin to that registry, while we wait for the donation proposal proceeds? I don't want to cause confusion but think it could be helpful for folks who are searching for a Kotlin implementation of OpenTelemetry.
I can't see how that could hurt.
I'm looking to contribute to this project! Exciting stuff!
I've got server-side Java/Kotlin experience and a some good knowledge of the OpenTelemetry ecosystem.
Thanks very much @kvithayathil and @brendenehlers for offering to contribute! It will be amazing to get contributions from anyone who is interested.
@danielgblanco I think that means the donation proposal can continue to the next stage, given the last blocker was engineers from more organisations committing to contribute towards the project? We will of course continue to try and find other folks who are interested.
I'd be happy to attend the SIG and contribute to the project should it go forward. I work for SolarWinds doing backend development in Kotlin.
I'm very much in support of this effort, and interested in contributing where possible. Excited to see where this can go.
This is a great idea! I would also be keen to contribute to the project.
@breedx-splk has that port hasn't had an update in 1.5 years, and I've got back several pages of commits and haven't seen a non-dependency update for close to 3 years. It feels like a dead project and not something to show on the registry?
How do we typically handle dead or nearly dead projects, @danielgblanco ?
Would love to contribute to this effort. We are already using otel-android in production, with an internal small kotlin layer on top it, using coroutine contexts for span propogation which is sub optimal. Also want to see if we can solve this at library level.
Also very in support of this project, I don't have much Kotlin experience to contribute but we are currently wrapping the existing Java/Swift Otel SDKs in a thin interface layer to use it in KMP, so this seems much better and easier!
Would at least be up for being a guinea pig and testing the packages :P
I'd like to help out also! I have experience developing KMP/CMP apps or features(not in production).
I think this is a great initiative, and I’d be happy to help out if time permits.
Thank you all for your support! It's indeed great to see so many folks willing to contribute and help maintain opentelemetry-kotlin. With a majority of @open-telemetry/governance-committee members having approved, we're happy to move this donation forward!
Following our donation process, and considering this does not qualify as a small donation, the next step is for @open-telemetry/technical-committee to conduct due diligence. I'm unaware of current bandwidth, but we'll keep this issue updated.
After due diligence is done, I propose creating a project proposal. This can help to bring together those contributors above, listed as contributors to the project, under a set of short-term deliverables to bootstrap the new SIG and get the code onboarded. Any pending items off the back of the due diligence can also make it as part of that project deliverables. The idea will be that after the project is completed, opentelemetry-kotlin should be a healthy SIG moving to business-as-usual operation.
If I understand this right, the code contains not only an alternative opentelemetry API than the one in opentelemetry-java (which is more idiomatic for kotlin users) but also an entirely new implementation of that API (i.e a new SDK implementation). And users have the choice of using the kotlin API + opentelemetry-java SDK, or the kotlin API + kotlin SDK.
I understand the desire for a kotlin idiomatic API (though I'd argue it'd be better to add this idiomatic API to our opentelemetry-api-kotlin API extension library), but I don't understand re-implementing the SDK. If its possible to plug in the opentelemetry-java SDK to the idiomatic kotlin API, why would the collective opentelemetry community want the significant burden of maintaining two SDK implementations (which contain many non-trivial parts) where one could suffice? You might say that kotlin users want a kotlin idiomatic SDK just like they want a kotlin idiomatic API, but the user facing portion of the SDK is a rather narrow programmatic configuration interface, with the bulk of the interesting bits non user facing.
Taking a cursory look at the donation project, I see signs of the large SDK implementation burden in the form of key missing pieces:
- The OTLP exporters appear to support a fraction of the configurable parameters
- The OTLP exporters protobuf serialization is inefficient (compared to the handrolled version in opentelemetry-java)
- The OTLP exporters don't have multi platform support
- The other exporters are missing (console, zipkin, prometheus)
- The concept of trace sampling appears to be missing
- The alternative configuration interfaces (environment variables, declarative config) are missing
- The concept of context propagation across application boundaries appears to be missing (can't find anything related to TextMapPropagator or w3c trace state context propagation)
- There is no metrics SDK implementation
@jack-berg I may be misunderstanding so someone more knowledgeable can correct me, but the compatibility mode (which I believe is the Kotlin API over the Java SDK) wouldn't work for Kotlin Multiplatform if you're targeting a non-JVM platform like iOS.
The underlying implementation MUST be in Kotlin for it to be usable on iOS, etc
@jack-berg that's correct - the code defines an idiomatic Kotlin API. Library consumers can choose to use between 2 implementations of that API: one that wraps opentelemetry-java, or one that uses a pure Kotlin implementation. I've designed the compatibility mode so that existing opentelemetry-java exporters and other features can largely be used as-is, if folks wish to do so.
One key point is that Kotlin Multiplatform can run on non-JVM platforms and opentelemetry-java can't. For example, if someone wanted to instrument Kotlin code that was shared between their Android/iOS applications (or just write instrumentation using one technology), Kotlin Multiplatform would be the go-to solution.
Another motivation was an idiomatic API that avoided builders/concrete types/default implementations, and would be friendlier for mobile engineers.
The code certainly isn't complete right now but Embrace is happy to keep working on the implementation of those features, and other folks have offered to contribute. Looking at the background of some of the potential contributors it's also my hope that adoption could grow the OpenTelemetry community by introducing more folks from the mobile world, and won't necessarily draw existing resources away from the Java implementation.
Did a little more research into kotlin multi platform (apologies I'm new to this and naive) and it does indeed appear that something like this would be necessary for OpenTelemetry to support KMP. Its interesting, where at least some of kotlin's success is attributable to full java interoperability, KMP represents a shift away from that as any java library dependencies becomes a liability that needs to be replaced in the non-JVM platforms.
It still is a huge amount of effort. The need to support multiple platforms probably means the effort is greater than one SDK implementation (although there will be code reuse). All instrumentation would need to be rewritten to use the kotlin API. All plugins (i.e. everything from opentelemetry-java-contrib and more) that are relevant to kotlin would need to be rewritten. If KMP becomes very popular (there are some signs that its going that way), then this effort will be well spent.
Question @fractalwrench - again I'm naive about KMP, but have you given thought to something like providing a kotlin API, and providing multiplatform implementations of that API by leveraging the existing opentelemetry-java, opentelemetry-swift, opentelemetry-js projects respectively? Not sure if something like this is possible, and I can already think of some downsides.
@jack-berg a Kotlin API that wraps the existing opentelemetry projects is definitely something that we've considered. We did this for opentelemetry-java as the JVM is still the most common platform for Kotlin. This approach does solve the problem for folks who want to use the existing ecosystem of exporters/instrumentation for opentelemetry-java, which folks can use today from the Kotlin SDK.
There are likely going to be subtle differences between the Java/Swift/JS APIs that will make it quite tricky to wrap everything behind one Kotlin API. IMO developing & maintaining a Kotlin SDK with this approach would be more work (from the perspective of the Kotlin SDK), as it's N separate implementations rather than a single codebase with a few platform-specific bits. Having said that, I wouldn't be opposed to separate compatibility layers for non-JVM platforms if other folks did want to contribute this - it's not something Embrace is likely to contribute.
Another advantage of a pure Kotlin implementation is that the behavior of how telemetry is captured will diverge much less between platforms than wrapping existing SDK implementations.
I will conduct the due diligence for this donation proposal on behalf of the TC. Will provide a summary next week.
I don't know if this is the right place to comment, but I would like to contribute to the project! I've been writing Kotlin for many years now, and my experience can be helpful. One thing I was considering is writing a K2/IR compiler plugin that could bridge the JavaAgent gap! I think we won't get 1 to 1 result, but we could get something different that, in my opinion, can be just as good!
I am also very much in favour of making the codebase more "Kotlin"- friendly. For example, I was playing around with context receivers, which could allow us to build even nicer/cleaner code!
