Lack of Subject-Object Distinction in Test Suite Events Prevents Linking to Commit Under Test

[mrmaxsteel]

We’ve been working to integrate cdevents.test.suite.finished.v1 into our CI/CD observability pipelines and wanted to raise an issue around semantic modeling that limits our ability to fully describe the intent and scope of the event.

Problem

According to the current spec, the subject field of a test.suite.* event is expected to represent the test suite itself (which makes sense). However, there is no first-class way to represent what the test suite was executed against—i.e., the system under test. In most CI systems, this is a specific commit in a code repository.

At present, this information appears to have no standard location in the event schema (please correct me if I'm mistaken!) and is relegated to customData (which makes it non-portable and hidden from tooling that only interprets the base spec).

Moreover, the testSuite.name field lacks semantic rigor—it does not capture what kind of tests were run (e.g., "unit", "integration", "system"). There is no recommended vocabulary, structure, or field for test category.

Proposal

We see two possible improvements:

1. Introduce the concept of an object (or target) to the event schema.

   • The subject is the actor of the event: the test suite.

   • The object is the system being tested: e.g., a Git commit.

   • This would align with subject-verb-object grammar and make the semantics clearer and more powerful.

   • Example:

     "subject": {
       "id": "unit-tests-123",
       "type": "testSuite",
       "source": "https://github.com/org/repo"
     },
     "object": {
       "id": "abc123def456",
       "type": "commit",
       "source": "https://github.com/org/repo"
     }
     

2. Standardize testSuite metadata more rigorously

   • Consider adding a required type or category field to distinguish between different classes of tests (unit, integration, system, security, etc.).
   • Possibly move toward a vocabulary or schema that aligns with existing test reporting formats like JUnit or TAP.

Why This Matters

Without a standard way to link a test run to the code it tested, downstream consumers of the event (e.g., dashboards, analysis tools) cannot easily compute lead time, test reliability, or attest Git commits with results. All of this limits the power of the CDEvents ecosystem to support DevOps and platform engineering use cases.

We're happy to contribute to the spec if this sounds aligned with its goals. Thank you for your work!

[afrittoli]

Thank you, @mrmaxsteel, for your interest in CDEvents and your proposal.

I agree, the ability to link events is crucial. In v0.4, we introduced the concept of links, which is meant to help express relationships between events. For instance you could link the testing event to a change submitted event to express the fact that a particular test was triggered by a specific change. You can read more about links inhttps://github.com/cdevents/spec/blob/v0.4.1/links.md.

I appreciated that a link to a change event is not necessarily the same as saying which version is being tested, since it could be for instance that the git sha being tested is the result of an automatic rebase of the code on top of main or other branches. So I think there is definitely space for improving the semantics as you suggested.

Not all event types would benefit from an object, so I feel that the target commit would be better placed within the subject, e.g.

  "subject": {
    "id": "myTestSuiteRun123",
    "source": "/event/source/123",
    "type": "testSuiteRun",
    "content": {
      "environment": {
        "id": "dev",
        "source": "testkube-dev-123"
      },
      "testSuite": {
        "id": "92834723894",
        "version": "1.0",
        "name": "Auth TestSuite"
      },
      "change": {
         "id":  "abc123def456",
         "source": "https://github.com/org/repo"
      },
      "trigger": {
        "type": "pipeline"
      }
    }

...but I'm curious about what other think about this @xibz @davidB @e-backmark-ericsson

WRT the test types, today we have https://github.com/cdevents/spec/blob/f95df21d7e3045a37d2c85e07f90805130fd65be/schemas/testcaserunstarted.json#L137-L147 - we can add more if needed.

[mrmaxsteel]

Thanks for the prompt response @afrittoli!

I think I probably agree not all events would benefit from an object - I like your proposal to store within the subject, that should certainly work.

Is change the correct identifier for an individual commit? I wonder if simply calling it commit(or revision if we want to be SCM agnostic) removes ambiguity and encourages the notion of referring to an explicit revision being tested, rather than a moving target like a PR or branch?

Ah, I hadn't seen the test types, that should certainly suffice!

[davidB]

Hi @mrmaxsteel,

Linking & correlating events is one of the main topics we should fix/clarify.

• We need to clarify how to identify a subject and how to reference it in a non-ambiguous way (eg, I'm for merging source & id into uri link, see #252)

• Currently, I don't use the links section, but information from the content section (mainly package & environment for my use cases), again, the spec should be more consistent (content has a closed definition, links has an open definition)

• About how to identify SUT there are various possibilities: code source, package, service,... So, as you suggested, a "type" like field should be needed, but it will introduce another way to reference a "subject" (already too many see #252 again).

• About what triggers the test (as in the example from @afrittoli), IMHO, it should not be part of the event

  • There are too many possibilities: a SCM push, a schedule, a deployment, an event X, ... see every possible trigger for Github/GitLab workflow, Jenkins, Tekton, Argo Events & Argo Workflow,...
  • The trigger could be unknown. Imagine if the events are generated from the test system (from the summary report or by the reporter)
  • If the trigger is defined, it should be possible for every kind of cdevent.