Would it be possible to have multiple [Property] attributes on a single xunit test method?

[iskandersierra]

I'm thinking on the case when I detect an issue in one of my property tests, I usually copy the seed and add a new method OriginalTest_Regresion2 just to test that seed in the future, but the method just have to call the OriginalTest with the same parameters.

[MyProperty] // Inherits from PropertyAttribute to add some Arbitraries
public void OriginalTest(SomeTestType data, OtherTestType request)
{
    // ...
}

[MyProperty(Replay = "1234,5678")]
public void OriginalTest_Regression1(SomeTestType data, OtherTestType request)
{
    OriginalTest(data, request);
}

[MyProperty(Replay = "5678,1234")]
public void OriginalTest_Regression2(SomeTestType data, OtherTestType request)
{
    OriginalTest(data, request);
}

What I'm asking is the possibility to do the following:

[MyProperty]
[MyProperty(Replay = "1234,5678")]
[MyProperty(Replay = "5678,1234")]
public void OriginalTest(SomeTestType data, OtherTestType request)
{
    // ...
}

I know that xunit could impose some restrictions on having multiple test discoverer attributes on a single method. I'm just asking to know if it could be possible somehow.

PD: After all, InlineData on Theory does something similar to what I would need. Maybe more like:

[MyProperty]
[PropertyVariation(Replay = "1234,5678")]
[PropertyVariation(Replay = "5678,1234")]
public void OriginalTest(SomeTestType data, OtherTestType request)
{
    // ...
}

Just thinking outloud

[kurtschelfthout]

Yeah not sure how it would integrate with xunit but probably something that can be worked out. First step would be to remove AllowMultiple = false and see what breaks :)

https://github.com/fscheck/FsCheck/blob/master/src/FsCheck.Xunit/PropertyAttribute.fs#L111

[ploeh]

I've been thinking about a feature like this myself, but what might even more be useful would be an ability to run only the failing test case.

AFAIR when you use the Replay property, you rerun all test cases (by default 100).

Now imagine a property that runs 100.000 test cases instead of 100. In one particular run, the framework may find a counterexample at the 97.327th run.

You'll now need to rerun all 100.000 test cases (or, at least 97.327) again with the Replay property...

Or do I remember it wrong?

[kurtschelfthout]

@ploeh you are correct, that is the status quo for v2.x.

In 2.x it's hard to implement because it needs an non-backwards compatible API change (or at least the obvious way to do it needs that, there's probably workaround involving global state...)

In 3.x this feature exists (i.e. you can run just one test) https://github.com/fscheck/FsCheck/blob/fscheck3/src/FsCheck/Runner.fs#L503

[iskandersierra]

I've been thinking about a feature like this myself, but what might even more be useful would be an ability to run only the failing test case.

AFAIR when you use the Replay property, you rerun all test cases (by default 100).

Now imagine a property that runs 100.000 test cases instead of 100. In one particular run, the framework may find a counterexample at the 97.327th run.

You'll now need to rerun all 100.000 test cases (or, at least 97.327) again with the Replay property...

Or do I remember it wrong?

I would like this too! I thought on using something like: MaxTests=100_000, Replay="1111,2222", Index=97_327. Then the Property engine could just skip the previous 97326 cases and call the method with the one that interest me. However I assume that the sequence of cases is dependent on the success/failure of previous cases, so it would not produce the same case.

[iskandersierra]

Another option for the index, is to have an optional parameter on properties, like: int testCaseIndex, so that we can use conditional breakpoints on them.

[kurtschelfthout]

Going to mark this as closed, since in v3 it tells you:

Last step was invoked with size of 2 and seed of (7852419065498467916,13949927723544264963)

which is everything you need to know to run precisely that step via Replay.

[rynoV]

which is everything you need to know to run precisely that step via Replay.

@kurtschelfthout could you expand on how to use this feature?

I'm on version 3.0.0-rc3, and in my tests if I get something like

Last step was invoked with size of 7 and seed of (5292858709037839463,3148337649946224299)

I try to run the test with the xunit integration:

    [<Property(MaxTest = 1, Replay = "5292858709037839463,3148337649946224299", StartSize = 7, EndSize = 7)>]

And with this when I debug the test, I expect at the first breakpoint to see the shrunk data shown in the output of the failing test, but the first run has different generated data (it also doesn't match the pre-shrink failing data). Am I doing something wrong?

If I use the seed from the previous line, without startsize or endsize or maxtest

Falsifiable, after 52 tests (6 shrinks) (8296668269162686089,4727217723912314783)

It replays exactly the same, but then it's difficult to use a debugger

[kurtschelfthout]

https://github.com/fscheck/FsCheck/blob/master/src/FsCheck.Xunit/PropertyAttribute.fs#L80

Replay = "5292858709037839463,3148337649946224299,7"