json-schema-diff icon indicating copy to clipboard operation
json-schema-diff copied to clipboard

[Project idea] Find lack of diff coverage using fuzzing

Open untitaker opened this issue 2 years ago • 1 comments

We can employ cargo-fuzz or cargo-afl to find two jsonschemas that are not compatible with each other, but produce no diff by the tool.

afl produces a random bytestring. we

  1. parse that bytestring into a lhs schema, a rhs schema, and a payload that we should validate against both schemas
  2. if that parsing already fails, just pass the test
  3. if the payload validates against both schemas, pass the test
  4. if the payload validates against lhs but not rhs, AND json-schema-diff produces no breaking changes, crash

on steps 2 and 3, we will discard a lot of payloads. afl will complain about this and say that coverage does not increase. but it will likely still find a lot of cases, if running long enough.

untitaker avatar May 12 '23 17:05 untitaker

This fuzzer can probably be seeded with schema definitions, diffs and example data from sentry-kafka-schemas.

untitaker avatar May 12 '23 17:05 untitaker