Improved support for separate out directories

[aschlicht]

The ceedling project.yml file allows unique global compiler defines for each unit test file. (Referring to build options defines such as gcc's -D on the command line.) When a build option define is given, the current ceedling master branch outputs the .out file to a unique subdirectory under the build/test/out or build/gcov/out directory (or other directory for other tools). The .o files; however, are not output to a subdirectory. Unit tests may successfully run, but gcov coverage fails due to overwritten and mismatched .o, .gcno, and .gcda files.

The changes in this pull request cause the .o files to compile into unique subdirectories for each unit test file only when a special define is given for that unit test file. gcovr and reportgenerator are capable of merging coverage information for each file. We get a combined coverage report showing which lines are covered by running the set of unit test files.

Here is an example defines section in a project.yaml. We have three unit test files to compile our source file(s) with different -D build options.

:defines:
  # When this option is set to true, then the -D<test_name>
  # flag is added to the compiler build options.
  :use_test_definition: true
  :common: &common_defines
    - TEST
    - UNIT_TEST
  :test:
    - *common_defines
  :test_preprocess:
    - *common_defines
  :test_unitTests_defineOption1:
    - *common_defines
    - CUSTOM_DEFINE=1
  :test_unitTests_defineOption2:
    - *common_defines
    - CUSTOM_DEFINE=2
  :test_unitTests_defineOption3:
    - *common_defines
    - CUSTOM_DEFINE=3

Here is an example console output for two source files compiled with three different -D options. The console output shows separate coverage for that file for each unit test. Coverage is not combined on the console while it is combined in the gcovr and reportgenerator outputs. It is not combined on the console because I do not know a way to combine gcov files using gcov tools.

The test name line, such as test_unitTests_defineOption1, is not written to the console for files that were not compiled under a subdirectory.

test_unitTests_defineOption1
sourceOne.c Lines executed:14.15% of 106
sourceOne.c Branches executed:11.11% of 90
sourceOne.c Taken at least once:5.56% of 90
sourceOne.c Calls executed:0.00% of 7

test_unitTests_defineOption2
sourceOne.c Lines executed:8.49% of 106
sourceOne.c Branches executed:6.67% of 90
sourceOne.c Taken at least once:3.33% of 90
sourceOne.c Calls executed:0.00% of 7

test_unitTests_defineOption3
sourceOne.c Lines executed:14.15% of 106
sourceOne.c Branches executed:11.11% of 90
sourceOne.c Taken at least once:5.56% of 90
sourceOne.c Calls executed:0.00% of 7

test_unitTestsA_defineOption1
sourceTwo.c Lines executed:100.00% of 227
sourceTwo.c Branches executed:100.00% of 139
sourceTwo.c Taken at least once:92.81% of 139
sourceTwo.c Calls executed:100.00% of 58

test_unitTestsA_defineOption2
sourceTwo.c Lines executed:27.75% of 227
sourceTwo.c Branches executed:24.46% of 139
sourceTwo.c Taken at least once:12.95% of 139
sourceTwo.c Calls executed:24.14% of 58

test_unitTestsA_defineOption3
sourceTwo.c Lines executed:95.62% of 160
sourceTwo.c Branches executed:95.70% of 93
sourceTwo.c Taken at least once:87.10% of 93
sourceTwo.c Calls executed:92.86% of 42

[aschlicht]

A different pull request moved defaults from vendor\ceedling\plugins\gcov\config\defaults.yml to vendor\ceedling\plugins\gcov\config\defaults_gcov.rb. After calling ceedling upgrade <project_name> on an existing ceedling project, both defaults.yml and defaults_gcov.rb exist in vendor\ceedling\plugins\gcov\config\. The values in defaults.yml appear to override the values in defaults_gcov.rb, which causes a run time error because I modified defaults_gcov.rb.

How can we modify the ceedling upgrade command to delete an existing defaults.yml?

[beckerzito]

Hey @aschlicht just to let you know that I really enjoyed this feature! I have been looking for a way to generate coverage reports to the same module based on multiple define configurations (different test modules). It will help me a lot. Hope you can find all the answers and we can see this feature merged as soon as possible!

Thanks!

[CezaryGapinski]

@aschlicht Idea with separated output directories is great! I'm going to extend it to put other generated files, like mocks, preprocessed headers into separated directories because it should simplify defines dependency tracking, simplify current implementation and also help to build tests parallel.

[beckerzito]

@mvandervoord any plan to integrate that? This is a very useful feature!

Thank you!

[aschlicht]

@mvandervoord,

Is there anything you would like changed in this pull request and / or Cezary's pull request #598? I find these changes to be useful. Members of my project team at work do as well for customizing unit tests within our projects. If you like these changes, then I'd like to see them in a ceedling release because I can say they are vetted and can update our projects to use a formal release.

Thank you, Aaron

[mvandervoord]

@aschlicht (Aaron) -- thanks for the confirmation. I definitely fall behind on being able to validate some of these features and then they tend to go stale. I appreciate your validation of them! You are using both PR's mentioned?

[aschlicht]

@mvandervoord My team at work is using the changes in this PR. I just helped someone modify their project today with these changes, which is why I am revisiting this PR. They also needed a way to force all test outputs to separate directories. I can see making that change with another project.yml option in a future PR.

I have not tested the changes in Cezary's PR; however, I did code review them and they look good to me. I could test them if you'd like me to. Shall I?

Also, please take a look at my comment from May 18. The changes in this PR do not cause an issue when creating a new ceedling project, but there are issues when upgrading an existing project due to defaults.yml remaining after the upgrade. What do you think is the right solution?

[beckerzito]

Hey guys, any update on this feature @mvandervoord . I'm also needing this update at work, but we don't want to use versions under development. We need to be using the last stable release.

Any plan to integrate that?

Thanks

[aschlicht]

@mvandervoord,

Other firmware engineers at my company are still finding these separate out directories to be useful. Like beckerzito stated, it is challenging to roll out these changes to everyone when they are not part of the stable ceedling. Is there an aspect of this PR that concerns you or you do not want in your main branch? I could break the changes apart.

One such change is ensuring a gcov data file is paired with a gcov notes file to avoid coverage errors on files without methods. These changes are in gcov.rb.

Thank you, Aaron

[mvandervoord]

My hope is to roll these changes into the next release. I like this feature a lot (and honestly have wished it was the way it worked for a long time). It's going to be a bit of a challenge, though. This PR is tracking main at the moment. I need to refactor it to track the changes in the release candidate branch (which has a lot of change to support threading, etc)

[mkarlesky]

I am pleased to report that the latest version of Ceedling soon be released (1.0.0 — still known as 0.32 elsewhere) has full support for the idea at the heart of this PR.

Effectively, Ceedling now builds each test executable (corresponding to a single test file) as a mini-project. Each mini-project can have its own defines and compilation/linking flags configured through matchers for test files. Test files can also have their own header search paths by way of a special macro used inside the test file. Each test executable's build generates artifacts to its own subdirectory apart from all others. This means that you can now have different unit testing conditions for the same source file as well as different coverage reporting results.

If you would like to test and run coverage on the same source file under different build conditions, you only need to create multiple test files exercising it and configure the build with the symbols, etc. you want for that test executable build.

See the extensively updated documentation, particularly CeedlingPacket.