testsuite
testsuite copied to clipboard
[Feature] Documentation rework
Proposal is to join all files that contain test-related documentation into one file with fixed formatting that will contain all info for developers and users. Currently, test-related documentation is located in these files: LIST_OF_TESTS.md, RATIONALE.md, USAGE.md, TEST-CATEGORIES.md.
Issues with current documentation:
- Hard to navigate (to get complete information about the test you need to browse multiple files)
- Redundant information (links to other files, repetition of test details)
- Hard to maintain (changes for one test result in need for change in multiple documentation files)
- Hard to verify (some or all info about test might be missing, it's time-consuming to validate that all documentation is present)
Proposals:
- Join documentation into TESTSUITE-TESTS.md file with fixed structure and formatting.
- Create GitHub pipeline that will validate that file and ensure that structure isn't broken and all needed sections are present
In addition to fixing current issues, new format for documentation will have these possibilities in future (not in the scope of the issue):
- Usage of documentation in code, directly parsed from the documentation. For example "remediation" in case if a test fails.
- Moving content of points.yml, including tags and points, to the TESTSUITE-TESTS.md, and using this in the code
- Possible automated generation of specialized documents, that will contain only subset of information for tests.
About having separate file with reasoning for users: is it really needed? If documentation is nicely structured, then it would be easy for user to find all desired information including reasoning for the tests.
Proposed structure of the documentation:
Category 1
Some info about the category
Test 1
Overview:
(task_name) Description of the test, what does it do
Reasoning:
Reasoning for the tests, why is it needed, how does it help for cloud-nativeness
Remediation:
What needs to be done if the test is failing.
Usage:
Command for running the test, extra dependencies if present.
Test 2
...
WIP: example of the file https://github.com/kosstennbl/cnf-testsuite/blob/test-doc-unification/docs/TEST_DOCUMENTATION.md