Use @pytest.mark.parametrize when generating pytest test cases

[kbaikov]

The currently generated tests have a lot of repeated code. Is it possible to just generate a list of tuples of input data and expected result that are usable by pytest parametrize mark decorator?

Let me demonstrate using the example from the documentation:

def test_case_1():
    var0 = -2603
    var1 = module0.triangle(var0, var0, var0)
    assert var1 == "Equilateral triangle"
    var2 = 1272
    var3 = module0.triangle(var0, var2, var2)
    assert var3 == "Isosceles triangle"

This can be rewritten using pytest.mark.parametrize:

@pytest.mark.parametrize(
    "a, b, c, expected",
    [
        (-2603, -2603, -2603, "Equilateral triangle"),
        (-2603, 1272, 1272, "Isosceles triangle"),
    ],
)
def test_case_1(a, b, c, expected):
    assert module0.triangle(a, b, c) == expected

This can be even improved if testdata would be declared separately as a list of tuples and later used once in the tescase decorator. See more examples here: https://docs.pytest.org/en/6.2.x/example/parametrize.html#paramexamples

It would be nice to also have test case names or ids to reflect what area of testing (e.g. boundary-value analysis or some edge case). Perhaps that is more of a n issue for the algorithm used to generate test cases. But i am cramming all this in one issue, which is not right.

For me personally the pytest parametrize usage would instantly upgrade this tool from academic interest to imediately usefull tool.

[stephanlukasczyk]

Dear @kbaikov ,

Thank you for your interest in Pynguin and for bringing this up.

I agree with your example that the test would be much cleaner and more readable when using the parameterize decorator from PyTest. However, I am not sure whether this easily generates. For the triangle example function, a parameterised test is much cleaner, but I have some doubts that this generalises to a general subject under test.

For me, this actually brings up a couple of issues that could (and should) be addressed:

• Reasonable naming for test functions and variables. Currently, Pynguin only names test functions and variables using a prefix and an incremented counter number. A general strategy to synthesize names seems to be hard, although there is research on this, for example this paper from ISSTA conference 2017. This is something, we might explore in a future version of Pynguin, it is on the future-features list.
• Minimisation of test cases. Currently, a test case is basically a collection of statements that get cut at a certain point. When writing tests manually, one usually follows some simple principle like “setup, execute, check” to keep tests simple. This is yet another feature on my future-features list, however.
• Generating parameterised tests. Finally, for cases as the one described by you, a parameterised test might be a nice and clean solution. There actually exists work in the context of test-generation for Java, see for example this ISSTA paper from 2011.

So, there is a whole bunch of work to do for us. Allowing to have all these techniques integrated into Pynguin seems reasonable, but unfortunately, it will take time until we can get there.