Improvement: Implement test dependency attribute(s)

[tregubovav-dev]

Dependency Requirements for Test Execution

Overview: There needs to be a clear dependency structure in place that ensures specific tests (referred to as Dependent tests) run only after their associated tests, test fixtures, or test categories (known as Dependee tests) have been executed. This structure should guarantee that Dependee tests are always run prior to Dependent tests, regardless of whether the Dependee tests meet the specified filter criteria.

Problem Statement: In my project, I utilize singleton classes that are initialized on demand. There are multiple fixtures within the unit test project; some of these utilize the singleton instance, while others do not. Additionally, I employ FastMM for memory leak monitoring in unit tests, which leads to a few common issues:

1. I must pay close attention to the naming of fixtures because I cannot control the order in which they are executed. Fixtures that depend on a specific singleton must always run after the fixtures that test that singleton. Otherwise, false-positive memory leak reports may occur because the singleton instances do not release memory following initialization—this behavior is by design.

2. Using filters can result in false-positive memory leak reports, particularly when the fixtures that test the singleton(s) are excluded from the test run.

Possible Solutions: To address these issues, I propose introducing an attribute or attributes that inform the test runner to include and execute specific tests (Dependee tests) before running the dependent tests. This could be achieved with the following attributes:

• DependOnTest(TestPath): Points to a specific test dependency.
• DependOnFixture(AFixtureClass): Points to a test fixture dependency.
• DependOnCategory(CategoryName): Points to a set of tests or fixtures that belong to a specific category.

Furthermore, it would be beneficial if a single test or fixture could declare multiple dependencies.

[vincentparrett]

I will need to think on this a bit, but my first instinct is "won't do" - unit tests should not be dependent on the order they are run, that goes against the principle of unit tests being "units of test". If your tests have dependencies, use the [SetupAttribute] to run the setup before each test.

[tregubovav-dev]

Thank you for your prompt response, Vincent!

I mostly agree that tests should not rely on the order of execution. However, some architectural patterns can introduce such dependencies.

In my case, there is a workaround: I can split a single unit test project into two or more separate projects. One project can focus on testing Singletons independently of their consumers, while another project can test the Singleton consumers, ensuring that the Singletons are explicitly initialized when needed.

I created this issue to see if anyone else is interested in this functionality.

[vincentparrett]

I would recommend using Spring4D - that way you can still use singletons but build the container in the setup so the singleton is isolated in each test.