Code coverage isn't correct

[wspr]

trafficstars

Maybe @ranford can help me here :)

For one test

I've added a choice = 3; option in runtestsuite.m which runs the test suite on ONLY testcuboidtorque03.m and produces coverage-debug.xml. Comparing the output here for private/cuboid_torque_z_z.m, there are lines in the coverage XML like:

              <line hits="384" number="82"/>
              <line hits="4" number="84"/>
              <line hits="4" number="85"/>
              <line hits="4" number="86"/>
              <line hits="384" number="89"/>
              <line hits="4" number="90"/>
              <line hits="4" number="91"/>
              ...

which match up with the edge case calculations:

        if any(a)
          Ex(a) = 1/8*w(a).*(-w2(a)-2*Cw.*w(a)-8*Cv.*abs(w(a))+w(a).*(2*Cw+w(a)).*log(w2(a)));
          Ey(a) = 1/8*w(a).*(+w2(a)+2*Cw.*w(a)+8*Cu.*abs(w(a))-w(a).*(2*Cw+w(a)).*log(w2(a)));
          Ez(a) = 1/4*(Cu-Cv)*w2(a).*log(w2(a));
        end

These are explicitly tested in this test, hence the 4 hits on such lines.

For all tests

But if I change to choice=2; in runtestsuite.m the test suite runs on all tests, and produce coverage.xml — this is the file that is linked to CodeCov. After running all tests I receive:

              <line hits="192" number="82"/>
              <line hits="0" number="84"/>
              <line hits="0" number="85"/>
              <line hits="0" number="86"/>
              <line hits="192" number="89"/>
              <line hits="0" number="90"/>
              <line hits="0" number="91"/>

Clearly something is going wrong here; shouldn't the hits per line accumulate as the entire test suite is run? More importantly, my coverage for this file is less than it should be!

[wspr]

For reference, this is the setup for creating the coverage file:

  import matlab.unittest.TestSuite
  import matlab.unittest.TestRunner
  import matlab.unittest.plugins.CodeCoveragePlugin
  import matlab.unittest.plugins.codecoverage.CoberturaFormat
  
  suite  = TestSuite.fromFolder('testsuite/');
  runner = TestRunner.withTextOutput;
  plugin = CodeCoveragePlugin.forFolder(pwd+["","/private"],...
    'Producing',CoberturaFormat('coverage.xml'));
  
  runner.addPlugin(plugin);
  results = runner.run(suite);

[acampbel]

Yes that does seem wonky. The first thing I'd suggest looking into is whether any of the code you are executing uses the profiler. This can invalidate the overall numbers, where a call to the profiler (while you are profiling to gather coverage) can result in losing prior coverage. Do you know if any of your code or any code you are calling is using the profiler?

[wspr]

@acampbel — no, pretty sure there's nothing like that. When I get a bit more time I'll try to break up the tests and see if I can reproduce the issue with a minimal example.

[wspr]

I tried to create a truly minimal example and couldn't reproduce this behaviour... I wonder if it's because the test cases are held in a separate folder. Snowed under again right now so will have to come back to this again later.