infix icon indicating copy to clipboard operation
infix copied to clipboard

Add framework for generating test spec + report

Open troglobit opened this issue 1 year ago • 1 comments

Nothing fancy, could be a README.md (markdown) in each test directory, or pydoc to extract the test's docstring in reStructured Text format.

Either way, how tests are documented should be standardized: description, topology, pass criteria, allowed deviations, i.e., reasons for SKIP, e.g., tests running on virtual hardware.

The goal is to be able to extract the documentation for each test and generate a test specification based on the test suite. Each test is numbered (e.g., section headings) and the test results should match those numbers in the test report. The spec and report can be different documents. Possibly everything is extracted to a test-result/ directory with sub-directories for each test so any type of format can be generated.

troglobit avatar May 13 '24 12:05 troglobit

Decision:

  • Go with Markdown, README.md for every test (it's what we use elsewhere)
  • generate topology with graphviz, as png for README.md, comitted to GIT
  • Further discussion on which sections should be included in README needed

troglobit avatar May 14 '24 09:05 troglobit

Decision from test description meeting:

  • One directory per test
  • Test specification should be generated from the code (checked in)
  • Each test should have a section for a textual description in docstring(?) format
  • Each test should have a logical topology in a .dot file, and an generated image that is checked in and included in the specification.
  • The test.step() should be used to auto generate the specification from the code, with something that is possible to do manually and pass criteria. example: "verify that it is possible to ping from host:data to dut1:data"

mattiaswal avatar Aug 26 '24 12:08 mattiaswal

We limit this task to generate the test specification. Let's create another (related) task for creating a test report.

jovatn avatar Sep 06 '24 08:09 jovatn