sway icon indicating copy to clipboard operation
sway copied to clipboard

Use fuzzing to generate Sway regression tests.

Open otrho opened this issue 3 years ago • 0 comments

We could either find a tool or write our own Sway source code fuzzer. In the past I've used AFL to generate files as input to algorithm tests and it was quite amazing what it discovered. I don't think AFL is able to efficiently build source code but there may be other similar tools out there which can.

The problem it solves is our testing is generated by humans who are already familiar with Sway and will make unconscious assumptions about source code, and in general not really push the compiler into the edge and corner case territory.

A fuzzer can randomly but also iteratively generate input for the compiler which can be sane, based on syntactic rules, but might not compile, or hopefully might even crash. Finding these bugs ourselves rather than by our users is highly desirable.

There are some examples in #2397 showing code which looks sensible but shouldn't compile, and which regular humans wouldn't usually write. This would be trickier to create with a fuzzer since it's hard to tell what should NOT compile even when syntactically correct, but still it's worth looking into.

otrho avatar Jul 28 '22 06:07 otrho