cpp-coveralls Build process needs to run twice to produce a working coverage report

I have fiddled with support for code coverage for our project https://github.com/GlPortal/RadixEngine/pull/30

However it only started producing proper results after I ran the build process twice https://github.com/GlPortal/RadixEngine/pull/30/commits/59a6968cc12718b0550809ac239d83041c15be15

Any idea what the problem is?

Here is a log of the build: https://travis-ci.org/GlPortal/RadixEngine/builds/212253813

Mar 10 '17 11:03 hhirsch

Strange, from your travis log I see 2 interesting sections

The first time coveralls only analyses one files, the second time it analyses 'a lot'.

https://travis-ci.org/GlPortal/RadixEngine/builds/212253813#L506

[100%] Built target tests
Test project /data
    Start 1: XmlHelperTest
1/1 Test #1: XmlHelperTest ....................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 1
Total Test time (real) =   0.01 sec

CMakeCXXCompilerId.gcda:cannot open data file, assuming not executed
File 'CMakeCXXCompilerId.cpp'
Lines executed:0.00% of 7
Creating 'CMakeCXXCompilerId.gcno##CMakeCXXCompilerId.cpp.gcov'

# it then moves on to doing other things

https://travis-ci.org/GlPortal/RadixEngine/builds/212253813#L715

[100%] Built target tests
Test project /data
    Start 1: XmlHelperTest
1/1 Test #1: XmlHelperTest ....................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 1
Total Test time (real) =   0.01 sec

/data/external/json11/CMakeFiles/json11.dir/json11.cpp.gcda:cannot open data file, assuming not executed

File '/usr/include/c++/6.3.1/bits/stl_algobase.h'
Lines executed:0.00% of 23
Creating '#data#external#json11#CMakeFiles#json11.dir#json11.cpp.o###usr#include#c++#6.3.1#bits#stl_algobase.h.gcov'

File '/usr/include/c++/6.3.1/bits/predefined_ops.h'
Lines executed:0.00% of 1
Creating '#data#external#json11#CMakeFiles#json11.dir#json11.cpp.o###usr#include#c++#6.3.1#bits#predefined_ops.h.gcov'

[...]
# lots more files analysed

Very strange.

Jul 06 '18 03:07 chrisosaurus

Attempted reproduction steps roughly following https://github.com/GlPortal/RadixEngine/blob/master/COMPILE.md (I tried to use a build dir at first but it didn't seem to add anything, so tried without)

$ sudo pacman -S cmake make gcc pkg-config assimp libepoxy sdl2{,_mixer} bullet mesa tinyxml2 freeimage
$ sudo pip install cpp-coveralls
$ git clone https://github.com/GlPortal/RadixEngine.git
$ cd RadixEngine
$ git submodule init
$ git submodule update
$ cmake .
$ make
$ make test
$ coveralls --dump HERE
$ grep -o '"name":[^,]*,' HERE | wc -l
890
$ grep -o '"name":[^,]*,' HERE | head -10
"name": "include/radix/Camera.hpp",
"name": "include/radix/SimulationManager.hpp",
"name": "include/radix/EntityManager.hpp",
"name": "include/radix/Entity.hpp",
"name": "include/radix/Console.hpp",
"name": "include/radix/SoundManager.hpp",
"name": "include/radix/BaseGame.hpp",
"name": "include/radix/World.hpp",
"name": "include/radix/GameWorld.hpp",
"name": "include/radix/Viewport.hpp",

so it seems to be correctly generating coverage report sfor 890 source files after a single run.

Are you still able to produce this? if so please provide reproduction steps.

Jul 07 '18 02:07 chrisosaurus

The test target does not exist so make test should just fail. make should build the tests though.

iirc coveralls not running wasn't the problem but it reporting 0% coverage when there is 3% coverage.

Variations can be easily tested on a branch and I'll be happy to change the line accordingly: https://github.com/GlPortal/RadixEngine/blob/master/.travis.yml

I tried to tackle this problem multiple times and I always had to revert to the run twice method.

Jul 07 '18 06:07 hhirsch

make test seems to run the tests for RadixEngine

[chris@niflheim RadixEngine]$ git remote -vv
origin	https://github.com/GlPortal/RadixEngine.git (fetch)
origin	https://github.com/GlPortal/RadixEngine.git (push)

[chris@niflheim RadixEngine]$ make test
Running tests...
Test project /home/chris/devel/RadixEngine
    Start 1: XmlHelperTest
1/3 Test #1: XmlHelperTest ....................   Passed    0.01 sec
    Start 2: QuaternionTest
2/3 Test #2: QuaternionTest ...................   Passed    0.01 sec
    Start 3: MathTest
3/3 Test #3: MathTest .........................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) =   0.03 sec

Am I misunderstanding something?

I guess I can try find time to setup my own coveralls and travis for a clone of RadixEngine and see what I can get out of it

Jul 08 '18 01:07 chrisosaurus

So in RadixEngine it looks like:

make tests will build the tests
make test and ctest are equialent - either one will run the tests

[chris@niflheim RadixEngine]$ make tests
[  3%] Built target RadixEntity
[  3%] Built target GLAD
[  4%] Built target json11
[ 11%] Built target VHACD_LIB
[ 33%] Built target angelscript
[ 36%] Built target easy_profiler
[ 88%] Built target RadixEngine
[ 93%] Built target MathTest
[ 96%] Built target XmlHelperTest
[100%] Built target QuaternionTest
[100%] Built target tests

[chris@niflheim RadixEngine]$ make test
Running tests...
Test project /home/chris/devel/RadixEngine
    Start 1: XmlHelperTest
1/3 Test #1: XmlHelperTest ....................   Passed    0.01 sec
    Start 2: QuaternionTest
2/3 Test #2: QuaternionTest ...................   Passed    0.01 sec
    Start 3: MathTest
3/3 Test #3: MathTest .........................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) =   0.03 sec

[chris@niflheim RadixEngine]$ ctest
Test project /home/chris/devel/RadixEngine
    Start 1: XmlHelperTest
1/3 Test #1: XmlHelperTest ....................   Passed    0.01 sec
    Start 2: QuaternionTest
2/3 Test #2: QuaternionTest ...................   Passed    0.01 sec
    Start 3: MathTest
3/3 Test #3: MathTest .........................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 3

Total Test time (real) =   0.03 sec

I've setup travis and coveralls for my clone, will play and around and then report results back, might take a while

Jul 08 '18 01:07 chrisosaurus

As a side effect of investigating this I noticed an issue in that you have a public .coveralls.yml (only needed for private travis, not needed for open source.)

I created an issue on your repo to describe this https://github.com/GlPortal/RadixEngine/issues/173 and then sent a PR to fix this https://github.com/GlPortal/RadixEngine/pull/174

That should just work as-is unless I am misunderstanding something.

Jul 08 '18 01:07 chrisosaurus

Okay, I now have coveralls running outside the docker container and I'm able to reproduce the 0% coveralls report on my clone.

Jul 08 '18 02:07 chrisosaurus

tests and output:

Running:

ctest inside docker ONCE
cpp-coveralls outside docker ONCE

gives 0% coverage Log: https://api.travis-ci.org/v3/job/401344880/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Running:

ctest inside docker ONCE
cpp-coveralls outside docker TWICE

gives 0% coverage Log: https://api.travis-ci.org/v3/job/401348185/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls --dryrun -i include -i source -e tests --gcov-options \'-lp\'
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Running:

ctest inside docker TWICE
cpp-coveralls outside docker ONCE

gives 0% coverage Log: https://api.travis-ci.org/v3/job/401348813/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Running:

'everything' inside docker TWICE
cpp-coveralls outside docker ONCE

gives 0% coverage Log: https://api.travis-ci.org/v3/job/401349052/log.txt

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'

Very strangely, only this last one produced a huge log. All previous versions produced

1.24s$ coveralls -i include -i source -e tests --gcov-options \'-lp\'
CMakeCXXCompilerId.gcno:version '603*', prefer '408*'
CMakeCXXCompilerId.gcda:cannot open data file, assuming not executed
File 'CMakeCXXCompilerId.cpp'
No executable lines
Removing 'CMakeCXXCompilerId.gcno##CMakeCXXCompilerId.cpp.gcov'
{u'url': u'https://coveralls.io/jobs/38190001', u'message': u'Job #11.1'}

only the last one (running cmake/make/ctest twice, coveralls once) produces the output we expect

[0K$ coveralls -i include -i source -e tests --gcov-options \'-lp\'
CMakeCXXCompilerId.gcno:version '603*', prefer '408*'
CMakeCXXCompilerId.gcda:cannot open data file, assuming not executed
File 'CMakeCXXCompilerId.cpp'
No executable lines
Removing 'CMakeCXXCompilerId.gcno##CMakeCXXCompilerId.cpp.gcov'

/home/travis/build/mkfifo/RadixEngine/CMakeFiles/RadixEngine.dir/source/SoundManager.cpp.gcno:version '603*', prefer '408*'
/home/travis/build/mkfifo/RadixEngine/CMakeFiles/RadixEngine.dir/source/SoundManager.cpp.gcda:cannot open data file, assuming not executed
File '/usr/include/c++/6.3.1/bits/stl_tree.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#CMakeFiles#RadixEngine.dir#source#SoundManager.cpp.o###usr#include#c++#6.3.1#bits#stl_tree.h.gcov'

File '/usr/include/c++/6.3.1/bits/alloc_traits.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#CMakeFiles#RadixEngine.dir#source#SoundManager.cpp.o###usr#include#c++#6.3.1#bits#alloc_traits.h.gcov'

[...]

File '/usr/include/c++/6.3.1/atomic'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###usr#include#c++#6.3.1#atomic.gcov'

File '/usr/include/c++/6.3.1/bits/atomic_base.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###usr#include#c++#6.3.1#bits#atomic_base.h.gcov'

File '/data/external/easy_profiler/easy_profiler_core/include/easy/profiler.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###data#external#easy_profiler#easy_profiler_core#include#easy#profiler.h.gcov'

File '/usr/include/c++/6.3.1/bits/locale_facets.h'
No executable lines
Removing '#home#travis#build#mkfifo#RadixEngine#external#easy_profiler#sample#CMakeFiles#profiler_sample.dir#main.cpp.o###usr#include#c++#6.3.1#bits#locale_facets.h.gcov'

{u'url': u'https://coveralls.io/jobs/38190033', u'message': u'Job #13.1'}

but it still seemed to generate a coverage of 0% unless something else went wrong.

I think the problem is at least in part due to your make process not producing coverage data the first time it is run, possibly due to cmake not correctly settings that variable you pass in.

I'll keep poking around.

Jul 08 '18 02:07 chrisosaurus

Confirmed that the single build isn't producing the profile data

        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l

Log: https://api.travis-ci.org/v3/job/401354549/log.txt

output

travis_time:end:014a2c6d:start=1531020384350424558,finish=1531020386367779703,duration=2017355145
[0K
[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.[0m
travis_time:start:12bf641b
[0K$ find . -iname '*.gcda' | wc -l
0

travis_time:end:12bf641b:start=1531020386374362869,finish=1531020386390520461,duration=16157592
[0K
[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.[0m
travis_time:start:03a20972
[0K$ find . -iname '*.gcno' | wc -l
1

running just ctest twice

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l

Log: https://api.travis-ci.org/v3/job/401354666/log.txt

outputs

travis_time:end:0557cb6e:start=1531020363849677574,finish=1531020365103355512,duration=1253677938
[0K
[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.[0m
travis_time:start:13f0336e
[0K$ find . -iname '*.gcda' | wc -l
0

travis_time:end:13f0336e:start=1531020365110018399,finish=1531020365127668255,duration=17649856
[0K
[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.[0m
travis_time:start:01fb3190
[0K$ find . -iname '*.gcno' | wc -l
1

running cmake/make/ctest twice

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l

Log: https://api.travis-ci.org/v3/job/401354576/log.txt

outputs

travis_time:end:0771da55:start=1531020637657257692,finish=1531020639721292131,duration=2064034439
[0K
[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.[0m
travis_time:start:1eb72f18
[0K$ find . -iname '*.gcda' | wc -l
54

travis_time:end:1eb72f18:start=1531020639731115174,finish=1531020639750849314,duration=19734140
[0K
[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.[0m
travis_time:start:1a6234f4
[0K$ find . -iname '*.gcno' | wc -l
172

So the problem looks to be that your build tooling requires running cmake/make/ctest twice before it produces the gcno and gcda files.

Cpp-coveralls doesn't appear to be the problem here.

Jul 08 '18 03:07 chrisosaurus

So poking around in CMakeFiles/ with grep looking for 'coverage' we get

single run of everything

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l
        - grep -Rni 'coverage' CMakeFiles/* | wc -l

Log: https://api.travis-ci.org/v3/job/401359763/log.txt

output

travis_time:end:1060f428:start=1531023482629119561,finish=1531023484216188718,duration=1587069157
[0K
[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.[0m
travis_time:start:076beae1
[0K$ find . -iname '*.gcda' | wc -l
0

travis_time:end:076beae1:start=1531023484225581695,finish=1531023484243654484,duration=18072789
[0K
[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.[0m
travis_time:start:015e15a6
[0K$ find . -iname '*.gcno' | wc -l
1

travis_time:end:015e15a6:start=1531023484255969002,finish=1531023484273764591,duration=17795589
[0K
[32;1mThe command "find . -iname '*.gcno' | wc -l" exited with 0.[0m
travis_time:start:308de1c4
[0K$ grep -Rni 'coverage' CMakeFiles/* | wc -l
2

So no gcno, no gcda, and only 2 mentions of 'coverage'

running cmake twice

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l
        - grep -Rni 'coverage' CMakeFiles/* | wc -l

Log: https://api.travis-ci.org/v3/job/401359814/log.txt

output

travis_time:end:0ce384e0:start=1531023535870351025,finish=1531023537691723347,duration=1821372322
[0K
[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.[0m
travis_time:start:1e795360
[0K$ find . -iname '*.gcda' | wc -l
54

travis_time:end:1e795360:start=1531023537698709357,finish=1531023537714924357,duration=16215000
[0K
[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.[0m
travis_time:start:0f774228
[0K$ find . -iname '*.gcno' | wc -l
172

travis_time:end:0f774228:start=1531023537721351873,finish=1531023537738222833,duration=16870960
[0K
[32;1mThe command "find . -iname '*.gcno' | wc -l" exited with 0.[0m
travis_time:start:0dd3c07f
[0K$ grep -Rni 'coverage' CMakeFiles/* | wc -l
3

3 mentions of coverage, 54 gcda files, 172 gcno files

running 'everything' twice inside docker

      before_install: 
        - docker pull glportal/whale-gcc:coverall
        - pip install --user cpp-coveralls
      script:
        - docker run -it --rm -w /data -v $(pwd):/data glportal/whale-gcc:coverall bash -c "cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest && cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest;"
        - coveralls -i include -i source -e tests --gcov-options \'-lp\'
        - find . -iname '*.gcda' | wc -l
        - find . -iname '*.gcno' | wc -l
        - grep -Rni 'coverage' CMakeFiles/* | wc -l

Log: https://api.travis-ci.org/v3/job/401359864/log.txt

travis_time:end:0599ce52:start=1531023759612471574,finish=1531023761327501590,duration=1715030016
[0K
[32;1mThe command "coveralls -i include -i source -e tests --gcov-options \'-lp\'" exited with 0.[0m
travis_time:start:1f4625f9
[0K$ find . -iname '*.gcda' | wc -l
54

travis_time:end:1f4625f9:start=1531023761335501133,finish=1531023761353031077,duration=17529944
[0K
[32;1mThe command "find . -iname '*.gcda' | wc -l" exited with 0.[0m
travis_time:start:134c59f5
[0K$ find . -iname '*.gcno' | wc -l
172

travis_time:end:134c59f5:start=1531023761360645133,finish=1531023761376400031,duration=15754898
[0K
[32;1mThe command "find . -iname '*.gcno' | wc -l" exited with 0.[0m
travis_time:start:2519013c
[0K$ grep -Rni 'coverage' CMakeFiles/* | wc -l
3

Again 3 mentions of coverage, 54 gcda, 172 gcno files.

Jul 08 '18 04:07 chrisosaurus

So at the very least, your cmake config is buggy, as these two commands produce different outputs:

cmake -DCOVERAGE:BOOL=ON ./&& make && make tests && ctest

cmake -DCOVERAGE:BOOL=ON && cmake -DCOVERAGE:BOOL=ON ./ && make && make tests && ctest

Jul 08 '18 04:07 chrisosaurus

I have filed https://github.com/GlPortal/RadixEngine/issues/175 for the cmake config issue

Jul 08 '18 04:07 chrisosaurus

I'm still seeing a lot of noise in the genereated output, lots of mentions for system included files which will throw your coverage stats way off.

Jul 08 '18 05:07 chrisosaurus

cpp-coveralls cpp-coveralls copied to clipboard

Build process needs to run twice to produce a working coverage report

cpp-coveralls
cpp-coveralls copied to clipboard