Define the Minimum Viable Schema for the machine-readable end-of-test summary

[oleiade]

Summary

This proposal seeks to define a minimum viable schema for the new machine-readable end-of-test summary. The goal is to create a formal, versioned contract (e.g., via JSON Schema, cue, or any other formal definition technology fitting the bill) to serve as a solid foundation for the new feature, avoiding the pitfalls of the previous, unstructured format.

Problem

The existing machine-readable summary format has several downsides, most notably a lack of a clear, versioned schema. This has led to it being "softly deprecated" and has forced users into brittle workarounds, such as parsing the human-readable text summary for automation.

As we begin implementing the new end-of-test summary, we must build it on a solid foundation. The first and most critical step is to define a robust, versionable, and well-structured data format. Without an agreed-upon schema, any implementation will be difficult to coordinate and prone to the same issues as the previous format.

Proposal

This issue proposes that we define and agree upon a Minimum Viable Schema for the new machine-readable summary.

The primary outcome of resolving this issue will be a formal definition of the summary's structure. This schema will serve as the contract for the Go code that produces the summary and for the users who consume it.

Goals (What this issue will deliver)

• Define Top-Level Keys: Agree on the minimal set of top-level keys for the first version of the document.
• Define Top-Level Value Shapes: Agree on the data types and basic structure for the values of those top-level keys.
• Produce a Formal Definition: Create a formal schema file (e.g., using JSON Schema) that captures these decisions. This will enable code generation, validation, and clear versioning.

Non-Goals (What is out of scope for this issue)

• Feature Parity: This schema will not attempt to include all data from the human-readable summary or the old format. We are explicitly focused on the minimum viable structure.
• Implementation: The Go code required to generate the JSON output is out of scope and will be handled in subsequent issues.
• Nested Structures: Defining the detailed format for groups and scenarios, is out of scope. For now, we will define them as objects, with their internal structure to be defined in future issues.

Concrete proposal

TODO

Tasks and deliverables

1. Discuss and iterate on the proposed top-level schema in the comments below.
2. Reach a consensus on the final structure for the MVP.
3. Create a formal summary.v1.schema.json (or other schema technology) file representing and tracking the format.
4. Create an initial documentation page in the k6-docs repo that describes this new v1 schema.

Open questions for discussion

• Machine-readable end-of-test document version Format (TODO: link to issue)?
• Formal Language: Should we commit to using a formal definition language such as JSON Schema for the formal definition? Its wide tooling support makes it a strong candidate. Are there other alternatives (like Cue) we should consider? We have the Cog project at Grafana which should facilitate this kind of endaveours.
• Naming Convention: Should we standardize on camelCase for all keys as shown in the example?