Any combination of Params/ParamsSource/ParamsAllValues is broken

[YegorStepanov]

trafficstars

[DryJob]
public class Benchmarks
{
    [Params("a", "b", "c")]
    [ParamsSource(nameof(Source))]
    public string Input { get; set; }

    [Benchmark]
    public void Test()
    {
        if(Input is "ddd" or "eee") Thread.Sleep(700);
    }

    public static IEnumerable<string> Source()
    {
        yield return "ddd";
        yield return "eee";
    }
}

Method	Input	Mean
Test	a	702.3 ms
Test	a	710.1 ms
Test	b	706.1 ms
Test	b	714.9 ms
Test	c	707.2 ms
Test	c	707.1 ms

More interesting scenario is for Boolean:

[DryJob]
public class Benchmarks
{
    [Params(false)]
    [ParamsSource(nameof(Source))]
    [ParamsAllValues]
    public bool Input { get; set; }

    [Benchmark]
    public void Test()
    {
        if(Input) Thread.Sleep(700);
    }

    public static IEnumerable<bool> Source()
    {
        yield return true;
        yield return true;
        yield return true;
    }
}

Method	Input	Mean
Test	False	1.085 ms
Test	False	1.134 ms
Test	False	1.261 ms
Test	False	705.045 ms
Test	False	707.033 ms
Test	False	713.798 ms

Some notes:

• no [Params] or [Params(true)] changes all table's Input for true.
• [Params(false, false) doubles the number of rows in the table. (6 rows for ~1ms, 6 rows for ~700ms)

Change Source from true, true, true to true, false, true:

Input	Mean
False	1.119 ms
False	714.345 ms
False	1.277 ms
False	1.127 ms
False	713.454 ms
False	715.087 ms

Change Source to false, false, false:

It equals to true, true, true.

Input	Mean
False	1.175 ms
False	1.456 ms
False	1.232 ms
False	715.423 ms
False	713.352 ms
False	707.733 ms

Merry Christmas 🎄

[adamsitnik]

Hi @YegorStepanov

I am surprised that BDN is trying to make it work instead of just failing the benchmark validation.

@AndreyAkinshin what is your opinion on this? Should we add a mandatory validator that says it's not supported? Or try to make it work?

[AndreyAkinshin]

@adamsitnik I believe we shouldn't support such cases. I have added the corresponding validator as a part of #2280