Guarantee that an arbitrary produces at least the specified set of values during all runs

[artkravchenko]

💬 Question and Help

Hey @dubzzz, thank you so much for the framework! I use it a lot in both unit and end-to-end testing, and it's a cool experience.

I wonder whether it's possible to guarantee for an arbitrary like

fc.record({
  x: fc.boolean(),
  y: fc.boolean(),
  z: fc.boolean(),
  // some other arbitraries
  a: fc.integer(),
});

— to always generate both true and false for each x, y, and z?

If we run the test 100 times, we'd most likely generate these true and false. But in our case, we run end-to-end tests and limit the number of runs to 6, so from time to time we get only true or false for any of x, y, or z. Example on RunKit.

Is there a way to provide such guarantees using the current release of fast-check? If no, what do you think about such a feature and the ways of implementing it?

Thanks!

Potential solutions

Partial examples (requires changes)

The examples parameter looks similar to what we're going to achieve. It supports passing the values we explicitly want to test. But if we use examples, we need to pass the values for all "variables", not only for x, y, and z, but for a as well. We could call them complete examples, whereas what we want is something like partial examples:

examples: [
  [{ x: true, y: true, z: true }],
  [{ x: false, y: false, z: false }],
],

— and in these 2 predefined partial examples, get a resolved by the arbitrary.

It might mean the following: "for the 2 starting test runs, use the following subset of property and generate the rest subset". Instead of examples, the option might be named partialExamples.

Instead of numRuns, use shouldRunMore (requires changes)

The shouldRunMore could look like

type ShouldRunMore<TProperty> = (passedRuns: number, usedProperties: TProperty[]) => boolean;

How we could use shouldRunMore:

function getOptionsByVariable(properties) {
  const variables = {};

  for (const property of properties) {
    for (const [key, value] of Object.entries(property)) {
      variables[key] = (variables[key] || new Set()).add(value);
    }
  }

  return variables;
}

function isBooleanFullyTested(options: Set<boolean>) {
  return options.has(true) && options.has(false);
}

function shouldRunMore(passedRuns, usedProperties) {
  if (passedRuns < 6) {
    return true;
  }

  const { x, y, z } = getOptionsByVariable(usedProperties);

  return (
    !isBooleanFullyTested(x) ||
    !isBooleanFullyTested(y) ||
    !isBooleanFullyTested(z)
  );
}

This method seems less preferable than the partial examples since it increases the number of test runs (potentially a lot if we apply it to non-boolean values and increase the required set of values).

[dubzzz]

Hey @artkravchenko

I just quickly went through your question and the suggestion you made of partial examples seems pretty doable IMO. Maybe for a first version, it would be great to have partial examples as a separate option in addition to examples*. But anyway it seems feasible to integrate such feature in fast-check.

*Actually, here is the sequence of values generated by the framework today (see https://github.com/dubzzz/fast-check/blob/master/src/check/runner/Tosser.ts):

• ...values coming from examples if any
• generate value based on random generator using seed
• generate value based on random generator using seed and jumped once
• generate value based on random generator using seed and jumped twice...

In the context of partial examples we would need something like (lazyGenerate function in snippet above could maybe take an additional parameter - the partial example to complete 🤔):

• ...values coming from examples if any
• first partial example (or empty) using random generator based on seed to feel the gaps
• second partial example (or empty) using random generator based on seed and jumped once to feel the gaps
• third partial example (or empty) using random generator based on seed and jumped twice to feel the gaps
• ...

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.