Idea: Run-once for cleanup

[fdw]

In the documentation, it states that

for the moment the attribute run-once is not designed to be compatible with cleanup decorations.

Since I would need that, I have been thinking about how to implement this. And I think I found a way :)

In TestSuite.BuildTestCases(), groups are flattened into IEnumberable<TestCaseData>. That means that at this point, we lose the information which tests belong together, which is necessary for a one-time cleanup (and helpful for a one-time setup). Now, my idea would be to add artifical "tests" before and after the flattened group that contain first only the one-time setup and last the one-time cleanup.

As an example, take this XML

<group name="Group">
  <setup run-once="true">
    <tasks>
    [...]
    </tasks>
  </setup>
  <test name="Test A">
  <test name="Test B">
  <cleanup run-once="true">
    <tasks>
    [...]
    </tasks>
  </cleanup>
</group>

With my proposal, it would be flattened into something like this list of test cases:

• Setup-only "test" containing the one-time setup and nothing else
• Test A
• Test B
• Cleanup-only "test" containing only the one-time cleanup.

Then, these tests are fed to ExecuteTestCases as previously. The new, artifical tests can set up or clean up and not do anything else; the actual tests run as usual.

I think this might work, and it can even handle nested groups without problems. What do you think?

[Seddryck]

As far as I understand your suggestion, the issue is that a user could just select the "Test B" in the interface and the setup and cleanup will never be executed. Your solution sounds valid only if the user is selecting all the tests.

The potential solutions are a little bit more tricky because you need to ensure the cases where tests are not run in the expected order or only a subset of the tests are executed. WIll take a look this week-end fixing #663.

[fdw]

Yes, you're right :( I hadn't thought of that possibility, sorry. I would need that feature, so I thought it would be nicer if I also think about solutions when just opening a ticket.

However, feel free to close this if it doesn't help.

Thanks again!

[Seddryck]

To wrap-up the complexity for implementing this feature: NUnit is passing one by one all the tests that must be executed during a run. As such, NBi can check if it's the first time that a setup element need to be executed because the status of the setup instance is clear: it has already run or not. On the other hand, how could NBi know that this test is the last one using a cleanup element? The only way that I see is to execute all the cleanups at the end of the test-suite when I know that no other tests will still be executed. Have you got any other ideas?

[fdw]

The only way that I see is to execute all the cleanups at the end of the test-suite when I know that no other tests will still be executed. The problem with this is that later tests may rely on earlier tests to clean up after themselves. For example, test A writes some data into a table, the following cleanup removes that data, and test B can now rely on the table being empty/pristine. If all cleanup happens at the very end, that does not help in this scenario.

That is why I suggested pseudo-test only for setup/cleanup. These will then be executed at the right time. However, like you said, this will not work if one selects tests from the GUI.

The basic problem is that each group in NBi is equivalent to one TestSuite in NUnit, but one cannot create multiple TestSuites dynamically. Without this, we're stuck with having to guess when one group is done.