cds-dbs
cds-dbs copied to clipboard
Expand compliance test suite
Goal
The goal of this PR is to extend the compliance test suite with more scenarios. While also cleaning up the existing compliance tests into a structured method of testing.
Reporter
When a test throws an error with the following pattern it will be considered to be part of the compliance test suite and will be put into the compliance reporter result.
- The message starts with
#
- The message use the
#
character as a separator for category levels
This way it is possible to quickly generate a markdown file for a database service that precisely describes all the gaps that the database service has.
Example
Errors
# Type # Date # Does not support '1970-01-01'
# Expression # `>` # Does not consider 1 to be larger then 0
Report
- {database}
- Type
- Date
- Does not support '1970-01-01'
- Date
- Expression
-
>
- Does not consider 1 to be larger then 0
-
- Type
Specification
While extending the compliance test suite a lot of assumption on the specifications will be made. The goal is to have all the compliance tests reviewed and thereby a specification is gradually finalized. Including specification documentation as comments next to the test definition and when required links to external resources to provide full clarity. If we use an ISO or IEEE standard make sure to add a link to the detailed specification.