Does not appear to be uploading correct results to coveralls

[rhwood]

xcov 1.5.0 does not appear to be providing correct information to coveralls.

Note that for EasyPeasy, travis-ci shows ~97% coverage, but Coveralls shows 0%

I'm seeing the same on a project I'm starting to write unit tests for.

[Tornquist]

I'm seeing the same issue.

[joeboyscout04]

Any progress on this? Have the same issue.

[gyulavoros]

We have bumped into the same problem.

Slather doesn't seem to work with Xcode 11 and the new xcresult format (still working based on Coverage.profdata).

Xcov picks up xcresult files properly, but publishes 0% to Coveralls.

[martin-key]

I am getting the same issue.

[escakot]

Same here. Also getting the same issue. Anyone got this working?

[iRILLLL]

anyone know how to solve it?

[klein-thibault]

Same issue, I get accurate reporting from xcov but it shows 0% in Coveralls.

Edit: after some investigation, the report generated by Xcode and xcov is incompatible with the Coveralls API. Xcode gives you coverage function by function but doesn't provide a line-by-line breakdown. The code is xcov looks for lines but they don't exist, and the coverage array ends up empty.

This is the parsed code coverage file from Xcode:

{
  "targets": [
    {
      "name": "MyApp.app",
      "files": [
        {
          "coveredLines": 0,
          "coverage": 0,
          "functions": [
            {
              "name": "variable initialization expression of myFunction 1",
              "coverage": 0
            },
            {
              "name": "MyViewController.viewDidLoad()",
              "coverage": 1
            },
          ],
          "name": "MyViewController.swift",
          "executableLines": 67,
          "location": "/Users/myname/Project/MyViewController.swift"
        },

This is the xcov output:

{
  "service_job_id": null,
  "service_name": null,
  "repo_token": "[REDACTED]",
  "source_files": [
    {
      "name": "MyApp/MyViewController.swift",
      "source_digest": "1234",
      "coverage": [

      ]
    },
    {
      "name": "MyApp/MyViewController.swift",
      "source_digest": "1234",
      "coverage": [

      ]
    },

The coverage array being empty explains why everything shows up as 0% in Coveralls.

According to the Coveralls documentation:

coverage Array

An array of coverage data, with item at index 0 representing the coverage for line 1 of the source code.

Acceptable values for the array is an integer representing the number of times that line is covered, or null representing a line that is not relevant to coverage (such as whitespace or a comment).

It seems that you can send your Xcode report to Coveralls using xcrun llvm-cov export -format=lcov according to this comment in the Coveralls repo.

Regarding xcov, it would require an API change in Coveralls to take function coverage, and an xcov code change to not look at line coverage but function coverage.

[chrispomeroyhale]

@klein-thibault I wrote the comment you referenced, maybe I can help clear this up. Although my memory is hazy, and I don't have the complete historical context, essentially the Swift toolchain has moved from gcov (of the gcc toolchain) to llvm-cov for generating coverage reports. This shift in tooling has muddied the waters.

Although I don't use xcov I had initially considered going down the path you're considering: Creating another tool for converting llvm-cov reports in the format coveralls expects. There is already a mix of community tools and some of which are no longer relevant. I found that there is already a coveralls tool that uploads gcov data, and that llvm-cov has an export command that converts from its format to gcov. Note that this format is confusingly called "lcov" even though it's part of the gcc toolchain not the llvm toolchain.

[fermoyadrop]

I get an error when I run xcrun llvm-cov export -format=lcov:

$ xcrun llvm-cov export -format=lcov
llvm-cov export: for the --instr-profile option: must be specified at least once!

any ideas why?

[chrispomeroyhale]

I get an error when I run xcrun llvm-cov export -format=lcov:

$ xcrun llvm-cov export -format=lcov
llvm-cov export: for the --instr-profile option: must be specified at least once!

any ideas why?

I believe it's telling you to include the instr-profile option as an input to the command. I have an example usage of this command in datapackage-swift which I'm assuming still works.

[fermoyadrop]

Thanks! I got it working. I'll leave it here incase someone else runs into the same issue:

$ xcrun llvm-cov export -format=lcov \ 
  -instr-profile /<derived_data_path>/<Your_App>-fcgfqeajxcgmylhgrapajzcluenu/Build/ProfileData/B12FDBE1-465C-4FD5-98C5-09B3B02B609B/Coverage.profdata \ 
  <derived_data_path>/<Your_App>-fcgfqeajxcgmylhgrapajzcluenu/Build/Products/Debug-iphonesimulator/<Your_App>.app/<Your_App> > coverage.info
$ coveralls-lcov --repo-token "YOUR TOKEN" coverage.info

Just to add another workaround, I've also found Slather, also available for fastlane. They accept profdata and parse the coverage field in the POST request to Coveralls correctly. They just support a handful of CIs, same as Coveralls, but I've opened a PR to support just repo_token, same as xcov