GenAI project and code organization in Python

[lmolkova]

trafficstars

We're trying to find the space for GenAI-related instrumentation libraries (related to the https://github.com/open-telemetry/community/pull/2326 project).

TL;DR: we need to host OTel gen-ai instrumentations somewhere in otel, we're going to evolve them along with semantic conventions, we need to ship them more frequently and with different versions than other contrib components.

Where instrumentations live

Based on the discussions in Python SIG, we have the following options:

Option 1. In a separate repo

Pros:

• Easy to have different release cadence / versioning
• Gives more autonomy to GenAI community

Cons:

• Harder to coordinate releases
• Could lead to duplication for GenAI-oriented database instrumentations and other areas
• Harder to have common code between genai and contrib instrumentations, add genai instrumentation to generic distro
• Harder to enforce common policies and make sure that genai instrumentations are aligned with the rest of the ecosystem

Option 2. In the contrib, manual release per-component

Pros:

• Resolves all the cons of Option 1
• May improve general contrib tooling and related instrumentations (e.g. HTTP)

Cons:

• Less autonomy
• Manual release is done by the maintainer (needs PyPI permissions) - it does not scale

Option 3. In the contrib, but as a separate group of instrumentations

• Needs some tooling changes to support different release schedule and versioning
• Needs some process changes to support per-component ownership along with the approval powers

---

I'd like to explore Option 3 since it can potentially satisfy all the needs.