Paramatised tests but not every combination

[Jab2870]

Hi,

I am quite new to munit and c programming in general so my apologies in advance if I'm missing something.

I would like something similar to paramatised tests but without necessarily running through every combination.

So, let's say I was testing a function called double.

I might want to call my test function with the following "parameters":

	int tests[][2] = {
		{ 1 , 2 },
		{ 2 , 4 },
		{ 0 , 0 },
		{ 20 , 40 },
		{ -5 , -10 }
	};

The first being the input and the second being the expected output.

The output might be something like this:

/example/something
  input=1                            [ OK    ] [ 0.00001201 / 0.00001016 CPU ]
  input=2                            [ OK    ] [ 0.00001104 / 0.00000970 CPU ]
  input=0                            [ OK    ] [ 0.00001222 / 0.00001034 CPU ]
  input=20                           [ OK    ] [ 0.00001271 / 0.00001039 CPU ]
  input=-5                           [ OK    ] [ 0.00001131 / 0.00001004 CPU ]

Thanks in advance

[Jab2870]

To add to this, currently I have a single test function that defines the array like above and runs all the test cases as a single test. This is not ideal as it is not clear which case fails

[codylico]

Outsider comment; feel free to ignore.

I can think of four options:

1. Assuming you are aware of the test parameter support: You could write a mini-parser that encodes your input-output pairs as strings, then parse the strings as a setup step:

static char* double_params[] = {
  (char*) "1 2", (char*) "2 4", (char*) "0 0", /* ... */ NULL
};

static MunitResult
test_double(const MunitParameter params[], void* user_data) {
  char* pair_string;
  char* endp;
  long int input, output;
  pair_string = munit_parameters_get(params, "pair");
  input = strtol(pair_string, &endp, 0);
  output = strtol(endp, NULL, 0);
  /* ... */
}

2. Write each "test case" as a separate actual test.

static MunitTest test_suite_tests[] = {
  { (char*) "/example/double/1", test_double_1, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
  { (char*) "/example/double/2", test_double_2, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
  { (char*) "/example/double/0", test_double_0, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
  { (char*) "/example/double/20", test_double_20, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
  { (char*) "/example/double/-5", test_double_m5, NULL, NULL, MUNIT_TEST_OPTION_NONE, NULL },
  // ...
};

3. Use the same test code, but different setup functions.

static void*
test_setup_1(const MunitParameter params[], void* user_data) {
  (void) params;
  return (void*)(&(tests[0]));
}
static void*
test_setup_2(const MunitParameter params[], void* user_data) {
  (void) params;
  return (void*)(&(tests[1]));
}
static void*
test_setup_m5(const MunitParameter params[], void* user_data) {
  (void) params;
  return (void*)(&(tests[4]));
}

static MunitTest test_suite_tests[] = {
  { (char*) "/example/double/1", test_double, test_setup_1, NULL, MUNIT_TEST_OPTION_NONE, NULL },
  { (char*) "/example/double/2", test_double, test_setup_2, NULL, MUNIT_TEST_OPTION_NONE, NULL },
  // ...
  { (char*) "/example/double/-5", test_double, test_setup_m5, NULL, MUNIT_TEST_OPTION_NONE, NULL },
  // ...
};

4. Use one test function that picks an input-output pair at "random", and tests only that pair. Then run that test many times and hope it hits all test cases. Since the test program always reports its random seed, you can re-run the test with the failure random sequence while you debug your code.

static MunitResult
test_double(const MunitParameter params[], void* user_data) {
  int input, output;
  int index = munit_rand_int_range(0, 4);
  input = tests[index][0];
  output = tests[index][1];
  munit_logf(MUNIT_LOG_INFO, "testing %i, %i", input, output);
  /* ... */
}

$ ./test_program
Running test suite with seed 0x12345678...
/example/double                      [ OK    ] [ 0.00001 / 0.00001 CPU ]
$ ./test_program
Running test suite with seed 0x0badc0de...
/example/double                      [ ERROR ] [ 0.0000152 / 0.0000152 CPU ]
Error: /home/someone/munit/example.c:64: assertion failed: my_output == output (-4321 == -10)
Error: child killed by signal 6 (Aborted)
$ gdb --args ./test_program --seed 0xbadc0de
#
# ... fix the code ...
#
$ ./test_program --iterations 100 --seed 0xbadc0de /example/double
Running test suite with seed 0x0badc0de...
/example/double                      [ OK    ] [ 0.001023 / 0.001022 CPU ]