Property based testing

[steve-chavez]

Problem

The regression on https://github.com/PostgREST/postgrest/issues/4109 and the fix https://github.com/PostgREST/postgrest/pull/4114/ proved that we're not always manually testing every combination of features on new changes.

Solution

Use property based testing with Quickcheck to ensure all combinations are tried.

I haven't used Quickcheck before but a rough idea is to generate ApiRequest values and have them those pass through the Plan and Query modules and check if executed queries are valid. Not necessary to go through the Response module.

Notes

• Good intro to quickcheck: https://begriffs.com/posts/2017-01-14-design-use-quickcheck.html

[steve-chavez]

At some point we had quickcheck tests: https://github.com/PostgREST/postgrest/blob/c9b2830e52184ad1abefa83619bb1b328213a14e/test/Unit/PgQuerySpec.hx

[jenstroeger]

I’ve been using hypothesis a lot which is inspired by/based on Quickcheck, and it too focuses on property based testing.

From my experience it’s great if the units/functions under test have certain properties (e.g. one is the inverse of the other, etc.) because then the test data can be handled transparently:

assert fn⁻¹( fn( data ) ) == data

But to just produce random test data it may not be well suited because you’d probably want to check results based on the generated input, i.e. you need some dependency between generated input and expected output? For that purpose I found a fuzzer (e.g. restler-fuzzer) or repeatedly drawing from a random producer (e.g. fakedata) to be a better option.

Mildly related you might find issue https://github.com/HypothesisWorks/hypothesis/issues/3614 interesting?

[steve-chavez]

I’ve been using hypothesis a lot which is inspired by/based on Quickcheck, and it too focuses on property based testing.

TIL. Thanks for the reference!

Note that the problem is not so much about data but about trying different combinations of query params, i.e. adding limit=10 toorder=headofstate.desc.nulls.last (this was https://github.com/PostgREST/postgrest/issues/4109). Which has something that our manual tests missed. The data could be static really.

The thing about property based testing is that it has to be fast to produce lots of combinations quickly. We can do this in Haskell as we wouldn't have to launch a web server + communicate through HTTP to validate our generated queries are correct (granted we'd have to use postgresql here but there might be a way around it), we can do this in memory only.

The only way we could use hypothesis would be if we expose our query generation through FFI (https://github.com/PostgREST/postgrest/issues/3932), which is possible too.. but more work.

[jenstroeger]

Note that the problem is not so much about data but about trying different combinations of query params, i.e. adding limit=10 toorder=headofstate.desc.nulls.last (this was #4109). Which has something that our manual tests missed. The data could be static really.

So… same input, different queries, same expected output?

[steve-chavez]

Sorry the late reply here, I've checked hypothesis and now I'm convinced it's the right tool for the job.

So… same input, different queries, same expected output?

For this particular case, it would be enough to check if the status code is a 2xx status and not a 500.

It would be a success if hypothesis can discover the bug on https://github.com/PostgREST/postgrest/issues/4109, using a postgREST prior to the change on https://github.com/PostgREST/postgrest/pull/4114.