Support generation of code coverage data

[r89m]

Tutorials

• https://piolabs.com/blog/insights/test-coverage-on-unit-testing.html
• https://blog.leon0399.ru/platformio-coverage-github-actions

---

Hi Ivan,

I searched for this kind of topic and didn't find anything but let me know if this is a duplicate.

I have my Transition library being tested on Travis using my ArduinoFrameworkFake library which has been working well since you gave me directions on how to import the library.

I'd now like to gather some information regarding coverage so that I can identify areas where I am not testing my code sufficiently. I tried using the build_flags section in platformio.ini to pass --coverage but was met with a lot of linker errors that I tried (unsuccessfully) to resolve. Here's a few of the combinations of options I tried using -g -O0 --coverage -fPIC -shared -Wl,-fprofile-generate -Wl,-shared -Wl,-fPIC -fprofile-arcs -ftest-coverage -fprofile-generate -Wl,-lgcov --coverage

I am successfully generating coverage using: https://github.com/r89m/Transition/blob/master/.travis.yml#L56 which is giving me the output I'd like (https://coveralls.io/jobs/22116632) but I'd love to be able to do something along the lines of pio test --coverage or similar and generate the necessary gcno and gcda files.

Any help would be appreciated.

Richard

[tanishqaggarwal]

I know this has probably fallen by the wayside by now, but here's a simple fix I found. Posting this here in case someone, like me, searches the ends of the Earth for how to configure this functionality but can't figure out how:

• Add --coverage as a build option
• Add an extra_script with the following content:

Import("env")

env.Append(
  LINKFLAGS=[
      "--coverage"
  ]
)

Works like a charm.

[tanishqaggarwal]

It's worth noting that I haven't been able to get coveralls to satisfactorily exclude random nonsense files from the coverage report, although this is probably more of a coveralls problem than a PlatformIO one. So far I've been trying:

coveralls -r . -b . -e "./test/output_export.cpp" -e ".pio/libdeps"

This invocation tries to print a test coverage for libdeps within the .pio folder as well. Not sure why.

[tanishqaggarwal]

See https://github.com/pathfinder-for-autonomous-navigation/CommonSoftware/pull/15

[XNinety9]

I'd love to see that feature added to the next release's roadmap ^^

[arduhe]

So do I :-)

[jcw]

With some web searches, I've been able to generate coverage data and visualise it (using lcov + genhtml), but one puzzle still remains: I'm not just running the app (natively, in Linux), I also want to include coverage data from all the test suites I've written. The data from these tests gets generated fine (in .pio/build/native/test/testname/...), but so far I've found it impossible to use it with lcov/gcov, because the test application binaries get overwritten: the final apps build with pio run and pio test all end up being the same executable, i.e. .pio/build/native/program.

Coverage data is really only meaningful when you can take the code though all its paces and then examine the aggregated reports. I suspect that anyone using pio test will run into the same issue.

In short: how can I generate and combine all coverage data, i.e. several runs of the real app as well as all test apps?

P.S. For reference, here is a trimmed version of my extra_script.py:

# Import("env") etc ...

def generateCoverageInfo(source, target, env):
    for file in os.listdir("verify"):
        call([".pio/build/native/program", "verify/"+file])
    call(["lcov", "-d", ".pio/build/native/", "-c", "-o", "lcov.info"])
    call(["genhtml", "-o", "cov/", "--demangle-cpp", "lcov.info"])

env.AddPostAction(".pio/build/native/program", generateCoverageInfo)

This works fine for the app, without the test suites, when launched as pio run -t debug.

[MBerka]

I don't have an answer for @jcw , but that script helped me produce the following that works with an existing pio test -e native test command. platformio.ini:

[env:native]
platform = native
lib_compat_mode = off
src_filter = +<*> -<*dirs*> -<to/> -<exclude/>
; basically lib and test folders
build_flags =
  -lgcov
  --coverage
extra_scripts = test-coverage.py

test-coverage.py:

import os

Import("env")

def generateCoverageInfo(source, target, env):
    for file in os.listdir("test"):
        os.system(".pio/build/native/program test/"+file)
    os.system("lcov -d .pio/build/native/ -c -o lcov.info")
    os.system("lcov --remove lcov.info '*/tool-unity/*' '*/test/*' '*/MockArduino/*' -o filtered_lcov.info")
    os.system("genhtml -o cov/ --demangle-cpp filtered_lcov.info")

env.AddPostAction(".pio/build/native/program", generateCoverageInfo)

[abanik96]

I don't have an answer for @jcw , but that script helped me produce the following that works with an existing pio test -e native test command. platformio.ini:

[env:native]
platform = native
lib_compat_mode = off
src_filter = +<*> -<*dirs*> -<to/> -<exclude/>
; basically lib and test folders
build_flags =
  -lgcov
  --coverage
extra_scripts = test-coverage.py

test-coverage.py:

import os

Import("env")

def generateCoverageInfo(source, target, env):
    for file in os.listdir("test"):
        os.system(".pio/build/native/program test/"+file)
    os.system("lcov -d .pio/build/native/ -c -o lcov.info")
    os.system("lcov --remove lcov.info '*/tool-unity/*' '*/test/*' '*/MockArduino/*' -o filtered_lcov.info")
    os.system("genhtml -o cov/ --demangle-cpp filtered_lcov.info")

env.AddPostAction(".pio/build/native/program", generateCoverageInfo)

Doesn't seems to work, ended up in compilation isse. Can anyone guide me from here??

[MBerka]

@abanik96 Hard to help without knowing what you actually wrote in the config file. However, it does look like you are trying to run in a different environment instead of the native plaform that I used, and PlatformIO's ESP32 libraries may not include gcov in the same way.

[abanik96]

@abanik96 Hard to help without knowing what you actually wrote in the config file. However, it does look like you are trying to run in a different environment instead of the native plaform that I used, and PlatformIO's ESP32 libraries may not include gcov in the same way.

@MBerka , You are correct did not use native environement, used the environment created by default in platform.ini while creating the project. Can you tell me the significance of this 'native' platform. Also, how can i include gcov in ESP32 platform?

[MBerka]

@abanik96 The native platform allows the C++ to be compiled and run on the same machine where PlatformIO is installed. I have only tried it on Ubuntu, but it probably works similarly on other systems with gcc. This makes the writing and testing of critical logic much faster. I do not know how to run the tests on ESP32 - I am doing the same thing as the creator of this issue, using a library to simulate Arduino framework functions on the dev machine.

[leon0399]

Btw I also added proper coverage calculation since @MBerka's solution only calculates coverage for tests themselves. I'm building firmware binaries with coverage flags also, but it required changing memory segments and collecting coverage with lcov --initial (it only requires gcno files)

I wrote an article on it, it is quite complex for single reply: https://blog.leon0399.ru/platformio-coverage-github-actions