Run <setup> block before <instance-settling>

[lukzas]

Currently, it seems that tasks inside <setup> run before actual test execution. I have a scenario where I need to run some setup before <instance-settling> block in order to properly generate TCs. Is there such a possibility?

Maybe there could be an attribute similar to run-once to override the default execution order? Or there could be an additional pre-execute block to be executed before BuildTestCases() is called?

[lukzas]

The only way I got it working is a following hack 😝

private IEnumerable<TestCaseData> BuildTestCases(IEnumerable<TestXml> tests)
{
    var testCases = new List<TestCaseData>(tests.Count());

    foreach (var test in tests)
    {
        ExecuteSetup(test.Setup, Variables); // Run setup before tests are built with instance variable
        [...]
    }
}

I've tried to do it better way by adding a new element to TestSuiteXml.cs of SetupXml type, adding it to TestSuite.xsd so it validates, calling ExecuteSetup in TestSuite.cs etc. Eventually, I successfully compiled the code and run tests but nothing happened in GUI, they were run as inconclusive and that's all... so it seems I fall short on the codebase knowledge. Maybe, you'd have an idea how to do it properly 🙏

[Seddryck]

I try to maintain a strict separation between the stage you read/discover the tests and the stage where you execute them. From this, I'm not planning to support, as such, the run (execution stage) of a setup before the instance-settling (discovery stage).

It doesn't mean that there is no option but I'd need more info on your case about why you want to do this. From my experience, I see two cases:

• You want to execute the setup only once for any test-case generated by the instance-settling. I'd recommend to put your instance-settling into a group and add the setup into this group with the run-once attribute. It's supported and genbiL is also supporting this kind of scenario.
• Before generating the test-cases, you want to execute the task that will influence how many test-cases are loaded. This is also supported, you need to use a custom implementation. More info on #487 Indeed in this custom implementation you can do whatever you want before generating the different instances.

I hope that you're into one of these use-cases. Keep me posted because I'll need to work on setup/cleanup as soon as I've time.

Important note: I received a message from a team member that run-once is not working in the last versions of NBi. I quickly checked and I think it's indeed the case. If confirmed, I'll fix this bug as soon as I've time. My time is really limited, probably until EOM.

[lukzas]

Thanks, I'll explore the options mentioned.

I try to maintain a strict separation between the stage you read/discover the tests and the stage where you execute them

Agreed, that's why I'm suggesting adding new prebuild or similar block code that behaves similarly to the current setup but executes tasks before discovery phase.

You want to execute the setup only once for any test-case generated by the instance-settling [...]

Yes, but I need to execute it before discovery phase. I'm using groups and run-once already.

[...] Indeed in this custom implementation you can do whatever you want before generating the different instances.

I was aware of that option, but it looks like overkill for my scenario as all the capabilities that setup currently have perfectly fit my needs, except they run after between discovery and execution phase In #584 it was also an option to write a custom extension that handles XML, but I've decided to enhance capabilities of current CSharp resolver instead for the same reason.

run-once is not working in the last versions of NBi

What is meant by that? Which version exactly? I'm using it and works just fine.

I hope that you're into one of these use-cases

I could use option two, but it is overkill IMO. I need to execute sql and cmd task before discovery or execution

[Seddryck]

Thx for the confirmation of your scenario. Indeed a custom extension is a bit overkill if you just have to execute one sql. Nevertheless, keep in mind that your prebuild tag could be hitten several times by NUnit, so be prepared to check if your prebuild has not already been previously executed in this prebuild task.

If you put a prepare or prebuild element in the instance-settling element, it would be fine for me. Up-to-you to implement it or I could do it somewhere in April. If you've questions about implementation, I could guide you.

[lukzas]

If you put a prepare or prebuild element in the instance-settling element, it would be fine for me

I doubt if it's gonna work for me as I have many tests grouped where each of them has a different <instance-settling> block. Then I'd need to duplicate the same piece of code. Also, I don't need to run it before each of them, but only at a group or suite level. I thought about having it next to setup block or even outside group next to the variables block.

Nevertheless, keep in mind that your prebuild tag could be hitten several times by NUnit, so be prepared to check if your prebuild has not already been previously executed in this prebuild task

I'm already doing that but at the script level. I guess at the code level I could reuse run-once check.

Up-to-you to implement it or I could do it somewhere in April. If you've questions about implementation, I could guide you.

I'm happy to try, especially with some guidance. However, I'm not sure I'll have time for it before April too 😩