test a malformed pattern

[karenetheridge]

trafficstars

I noticed we had no tests for bad patterns.

[awwright]

An invalid schema doesn't return either valid or invalid; the validity of the instance is indeterminate.

[karenetheridge]

@awwright obligatory: https://thedailywtf.com/articles/What_Is_Truth_0x3f_ :)

[karenetheridge]

Putting on hold until we can figure out how to test schemas that would cause an implementation to throw an exception / send up an error / do something other than return validated=true/false.

[Julian]

It's going to be more backwards compatible for users of the suite to put these kinds of tests in a separate folder (because one wouldn't know to reference the flag until after we add such a thing).

So there's seemingly little disadvantage to doing it in a separate folder which we can bikeshed on (neither-valid-nor-invalid-tests/); I think I once discussed some of this with @ssilverman and put some notes in a doc.

But yes we should decide how to handle that "once" and then do it here, agreed.

[awwright]

I don't anticipate many compatibility problems, depending on how the tests are written. Especially if the "tests" property is empty or missing, that should not pose a compatibility problem.

However, I think the most intuitive way to write this would be to use tests: null to indicate the schema itself is malformed:

[
    {
        "description": "enum range error",
        "schema": { "enum": 4 },
        "tests": null
    }
]