[Initiative]: Observability Query Language Standardization Specification

[manolama]

Name

Observability Query Language Standardization Specification

Short description

Define a set of models, querying semantics and operators to analyze observability telemetry data.

Responsible group

TAG Operational Resilience

Does the initiative belong to a subproject?

No

Subproject name

No response

Primary contact

Chris Larsen

Additional contacts

@alolita @halcyondude @vjsamuel

Initiative description

TAG OBS Query Standardization WG Charter

Summary

With the success of CNCF’s OpenTelemetry, observability instrumentation and collection has been standardized across the industry. Adopters are able instrument metrics, logs, traces and profiles in their systems one time and experiment with various observability backends or vendors by simply reconfiguring OTel collectors.

Unfortunately, every backend and vendor provides a different way to query and process the data. A plethora of domain specific languages (DSLs) exists with different telemetry models, semantic definitions and analytical capabilities. While pointing OTel at a new vendor is simple, rewriting dashboards, alerts and queries is not.

The goal of this working group is to recommend a set of standard models and query semantics for observability data. Such an open standard would unlock simple migrations using automated tooling, vendor solutions on top of customer provided data lakes, integration with business data, lower operational burden for observability engineers and reduce the cognitive load of observability data consumers.

Completed Work

1. Completed a set of DSL designer interviews, documented in GitHub. Languages include PromQL, TraceQL, DataDog QL, NRQL, KX’s Q, PPL.
2. Recorded and published presentations from DSL designers including PPL, KX Q, PromQL, NRQL, UQL, TraceQL, TScript, etc.
3. Documented a set of telemetry models in GitHub.
4. Documented a small set of observability use cases in GitHub.
5. Query language features rubric.
6. Draft semantic specification.
7. Draft context document surrounding the semantic specification.

Milestones (target H2 `26)

Work that is currently underway via biweekly meetings with end users and representatives from hyperscalers and observability vendors reviewing the standards. (videos):

1. Complete reviewing and commenting on the draft semantic specification.
2. Define and review standard observability specific operators

Follow Up/Concurrent Initiatives

The following are initiatives building on the deliverable of this work:

1. Reference implementation of an observability query federation and proxy project
2. Standard syntax recommendation
3. Blog posts about the spec
4. Maintenance of the standard

Resources

• CNCF Slack Channel: #obs-query-standard-wg
• CNCF Tag Observability Github Repo
• CNCF Observability Query Standard WG Meeting Notes
• TAG OBS Query Standardization WG Charter
• CNCF Observability Query Language Feature Rubric
• [CNCF DRAFT QLSWG Semantics Specification Context](https://docs.google.com/document/d/1A_r7tpLVELIhF1g6xnkmF3RNkX5_XWrSkEU0gKXKIK0/edit?tab=t.0
• CNCF DRAFT QLSWG Semantics Specification

Deliverable(s) or exit criteria

1. The final deliverable is a published observability query language semantic specification for modeling and querying and analyzing data.
2. A Github repo with examples, context and interview results.

[alolita]

Are there any updates on reviewing this initiative? This effort was an ongoing workgroup in the earlier TAG Observability. We would like to continue our momentum on this effort and complete the ongoing work. Please let us know on next steps.

cc @TheFoxAtWork @manolama

[TheFoxAtWork]

Per today's TAG OR call discussion - this effort this (draft specification) ~~may be eligible to become a standard via the JDF~~. JDF requires widespread adoption and viability for ISO standardization. It doesn't belong under the OpenTelemetry project because while OTEL heavily referenced, it is still agnostic to the project.

There are two potential pathways for this to take:

• keep this as an initiative and the TOC would need to define an alternative process for initiatives to become standards.
• convert the specification draft into a new CNCF project applied at the sandbox level, this would follow the TOC's initial outline for taking successful specification projects ~~and elevating them to a standard via JDF~~ .

Related: https://github.com/cncf/toc/issues/1744 ~~Prior notes on JDF pathway integration to the moving levels process -~~ ~~> (no decision made, just planning) Projects apply to the TOC, standards [sub-project/initiative] created when approved, TOC needs to define criteria for pursuing this (limited to graduated? incubating?)~~ ~~> Core items: the standards development sub-project is expected to use github public repos for standards development. We will need to define WHY a project would want to do this. Earlier integration of standards development is preferred for projects interested in achieving this with project specifications to reduce burden later on development.~~

[TheFoxAtWork]

Initial discussion with Chris A. suggests this be submitted as a project to the CNCF.

[alolita]

Thanks for the guidance @TheFoxAtWork @caniszczyk

We will re-submit as a project to the CNCF.