Filter test cases to run based on dependents of an entity

[joshrsmith]

A common scenario when developing is that once you update a block you would want to run the simulation of that block and all blocks that are dependent on it. This would be faster than running all simulations in a vunit project, focusing just on code that may have been impacted by the change.

I am thinking something like this: python run.py --dependents-only <entity_name> [tests]

I think all the other ways of filtering / including / excluding test cases would still apply, but this would first prune the list of test cases to those relevant to the specified entity.

Maybe the entity_name should be a list of entities?

[kraigher]

It is a good idea. I would expect it to work for packages as well.

[jondhoor]

Has this been implemented yet in some form or another?

[umarcor]

@jondhoor, not yet. There were some recent dialogues about how to programmatically filter the tests to be executed: #741, #740. You might follow a similar approach, along with using VUnit dependency and change scanning features, in order to implement it in the run.py file. That'd be a nice prototype to use as a foundation for the --dependents-only feature requested here.

[siziegler]

For my regression testing I am also looking for a function that when I change a source file VUNIT detects automatically which testbenches need to re-run. I guess it should be possible to detect based on the timestamps what files have changed and need to be re-tested?

At some point the timestamps are already evaluated since VUNIT only re-compiles the files that effectively have changed. However, I am not sure how I can get this information back into my Phyton script so that I can decide if the simulation need to re-run or not.

[jondhoor]

Thanks for the replies. I am currently working on an adaptation that restricts the test case filter, to only allow test benches that are dependent on a certain set of input files. For example, in dependency graph below, changing source file A will run TB0, while changing source file D will run both TB0 and TB1.

python run.py --dependent-on src_fileA --list Listed Test cases: TB0.TC1 TB0.TC2 TB0.TC3 python run.py --dependent-on src_fileD --list Listed Test cases: TB0.TC1 TB0.TC2 TB0.TC3 TB1.TC1 TB1.TC2 TB1.TC3

First tests seem promising, I'll update later with more info/results.

[umarcor]

FTR, https://github.com/VHDL/Compliance-Tests/blob/master/run.py shows a possible approach to override which sources to include, which tests to run and how to build a report "manually". That can be an inspiration for prototyping features to then be contributed to the codebase.

[umarcor]

I guess it should be possible to detect based on the timestamps what files have changed and need to be re-tested?

~~@siziegler that is done by VUnit already, by default. First time, VUnit will compile all sources. Then, it will only recompile the ones that changed, and the dependents on those. However, that does not prevent tests from running, because there might be reasons to execute them again despite sources not changing (e.g. because some external data changed). Hence, this issue is about reusing a knowledge available in VUnit already. Unfortunately, I am not familiar with the codebase that relates the file dependency tree to the logical dependency tree.~~

EDIT

I did not see the second paragraph in @siziegler's message. I'm sorry about that.

[jondhoor]

I have made an implementation of an extended filter in the VUNIT ui module:

Is this something that would be viable for integrating into the framework? This allows to easily simulate all tests that are dependent on one or multiple source files. It does not automatically take into account which files have changed, but if your files are integrated into a git workspace, it is very easy to retrieve the changed files.