Define test plan elements that can be changed without bumping version number -- the editorial content or metadata

[mcking65]

This issue is inspired by the conversation in #760.

Some test plan elements can be changed without the need to create a new version and thus do not have to go through the working mode review processes. For example, changing the presentation order of tests is purely editorial and has no impact on test results. Thus, making this kind of change to a test plan should not trigger the working mode review processes.

The remaining elements of the plan are substantive. Making substantive changes to a plan creates a new test plan version that needs to go through the working mode review processes.

The objective of this issue is to agree on and document which elements are non-substantive and which are substantive. Then, if necessary, making adjustments to which fields are hashed. If we make changes to which fields are hashed, that change should only impact new plan versions. All existing plan versions should remain as they are, even if a change would have made some of them unnecessary.

From issue #760 ...

[howard-e]

Content currently flows from the w3c/aria-at/tests folder to the aria-at-app database's TestPlanVersion table via built assets from CSV files and transformed JSON data. That means any updates to the /tests content must also be reflected in the aria-at-app database.

Proposed Non-Substantive Changes

The following content changes are considered non-substantive:

• assertions.csv
  • assertionStatement
  • assertionPhrase
  • refIds
• references.csv
  • refId
  • type
  • value
  • linkText
• scripts.csv
  • setupScriptDescription
• tests.csv
  • title [1]
  • presentationNumber [2]
  • instructions
• {AT}-commands.csv
  • presentationNumber [2]

@mcking65, do any of the above items need to be removed from consideration, or should any be added?

Proposed Test Plan Version Service Changes

Currently, there is a scheduled service that imports new TestPlanVersions from w3c/aria-at. It would be beneficial to extend this service (or trigger a separate one, given resource considerations) to check for only non-substantive content changes. This service would:

• Backup existing data for the identified pattern, for potential restores.
• Update (migrate) the content in-place for the latest version of the pattern with the non-substantive changes.

Therefore, the current import process wouldn't create a new version unless substantive changes are found. If substantive changes are found, it will continue to work as it does today.

Example Scenarios

1. Scenario 1: V24.03.05 for PATTERN_A is in DRAFT. A typo in TEST_1.title is found. An update is made on March 6, 2024. The service updates TEST_1.title across all pages where relevant but keeps V24.03.05 as the current version.
2. Scenario 2: Similar to Scenario 1, but the typo is in ASSERTION_1.assertionStatement for PATTERN_B. An update is made on March 6, 2024. The service updates ASSERTION_1.assertionStatement across all pages where relevant but also keeps V24.03.05 as the current version.
3. Scenario 3: For PATTERN_C, typos are found in both TEST_1.title and AT_COMMAND_1.settings. The service deprecates V24.03.05 as it does today, and the latest version of PATTERN_C is now V24.03.06.

Limitations and Considerations

1. This proposal applies only to the V2 Test Format Definition, not the V1 Test Format Definition.
2. If only non-substantive changes are made, will there ever be a need to update beyond the latest test plan version for the pattern?
3. Are there any non-substantive changes that should be considered substantive when the TestPlanVersion is in the RECOMMENDED phase?

---

Notes

• [1]: Uncertainty remains about whether tests.csv#title should be classified as non-substantive.
• [2]: Although this was the provided example, we will need to label it as a substantive change today. Automation uses it as a stable identifier during it's result reporting. We're already aware of this limitation and would like to resolve it. For now, a separate issue can be created to track resolving it which would then allow the support of it as a non-substantive change in the future.