Schema change / purl-test: Add validation test type

[mjherzog]

This issue is intended to raise the visibility and documentation of a proposed addition to purl-test-schema.json from its introduction in PR #614 which proposes a new test type for validation. The PR covers the proposed change to the schema and corresponding updates to the 32 registered PURL types.

The definition of a validation test_type in #614 is: " "validation": "A PURL validation test, checking if a PURL follows all the rules for a particular ecosystem." A better-stated definition is: "A test to validate that an input PURL string complies with the rules for its PURL type."

The proposal for the new validation test_type also adds the new property of a purl_validation_message to the PURL test schema for validation tests.

"purl_validation_message": {
      "title": "PURL validation message",
      "description": "Validation severity and message to use as a test expected output when validating a PURL.",
      "type": "object",
      "additionalProperties": false,
      "properties": {
        "severity": {
          "title": "Validation severity",
          "description": "Validation severity.",
          "type": "string",
          "enum": [
            "info",
            "warning",
            "error"
          ],
          "meta:enum": {
            "info": "Informational validation message",
            "warning": "Warning validation message",
            "error": "Error validation message"
          }
        },
        "message": {
          "title": "Validation message",
          "description": "Validation message.",
          "type": "string"
        }
      }
    },

The tasks to implement this change are:

[sjn]

This needs also a "success" severity.

[pombredanne]

@sjn re:

This needs also a "success" severity.

IMHO success is the default state, do you think we need this explicitly?

[sjn]

@sjn re:

This needs also a "success" severity.

IMHO success is the default state, do you think we need this explicitly?

Yes, because sometimes error conditions don't produce an output. If you expect an output and don't get it, you can derive useful information from this. (Though this only works if you are outputting anything useful beyond what is expected from the other severity levels)

For example: how do you detect if a test unexpectedly exits with a 0 return code?

You can detect this failure mode if you expect a final success output.