JSON-Schema-Test-Suite icon indicating copy to clipboard operation
JSON-Schema-Test-Suite copied to clipboard

idea for expected failure tests

Open gregsdennis opened this issue 1 year ago • 3 comments
trafficstars

This PR adds an idea for tests that check failure scenarios. Please see the included readme for details.

~(I'm pretty sure we have an issue for this. I'll come back and reference it, but I have look for it first.)~

Looks like this starts to address #589.

gregsdennis avatar Apr 06 '24 22:04 gregsdennis

Yeah, I ran through all of that same logic when putting this together.

gregsdennis avatar Apr 08 '24 20:04 gregsdennis

There's also a question that I don't think is addressed by the spec: if the error in the schema isn't exercised by the instance, is the implementation still required to error?

For example, a bad items keyword with an object instance.

I don't think that the spec says anything about this case. If it doesn't, then we really can't provide a test for it. Maybe we can add an optional/ subfolder here, too.

gregsdennis avatar Apr 08 '24 20:04 gregsdennis

I agree that the spec doesn't say anything about that case. I'm not worried about covering pathological cases, so I don't think an optional folder is necessary. I think providing instances that will exercise parts of the schema that have errors is good enough.

jdesrosiers avatar Apr 08 '24 23:04 jdesrosiers