Gather test coverage data for danger-swift

[yhkaplan]

Reading the Swift 5 release notes for Xcode 10.2 beta, it sounds like SPM projects will be able to gather test coverage data via the command line (my understanding is that it is not possible in current versions of Swift):

The swift test command can generate code coverage data in a standard format suitable for consumption by other code coverage tools using the flag --enable-code-coverage. The generated code coverage data is available inside build-dir/configuration/codecov.

https://developer.apple.com/documentation/xcode_release_notes/xcode_10_2_beta_release_notes/swift_5_release_notes_for_xcode_10_2_beta

I would like to make a Danger-Swift plugin to grab this coverage data and then use this to comment coverage info for each PR in this repo. (I hope "standard format suitable for consumption by other code coverage tools" means JSON.)

Sound good?

[f-meloni]

Yep, I was actually having a look to create a plugin that uses https://github.com/nakiostudio/xcov, will check if I can make it work with that as well.

[yhkaplan]

Oh, have you started working on the plugin or is it in the idea phase? I was thinking of using Apple’s similarly named xccov (extra c) tool now that it can output JSON as of Xcode 9.2 or so.

[f-meloni]

I started writing the plugin, but I'm just at the beginning (just started with the model etc), I can still change tool, will have a look if is better use xccov :) I remember when xccov came out I decided on porpouse to not use it and use xcov instead, but i don't remember why :D

[f-meloni]

Also I will evaluate Linux compatibility.

[yhkaplan]

Hmm, so this appears to be a completely different JSON format than that of xccov, which mostly just provides coverage percentage for each file, and a total. This seems to provide line number coverage, suitable for being used in an editor, as well as percentages for files. "filename" and "percent" under "summary" should be enough for now.

Here's the path where I found the file: .build/x86_64-apple-macosx/debug/codecov/{Package name}.json

{
  "version": "2.0.0",
  "type": "llvm.coverage.json.export",
  "data": [
    {
      "files": [
        {
          "filename": "/Users/joshk/dev/github.com/yhkaplan/Reg/Sources/Reg/Reg.swift",
          "segments": [
            [
              6,
              59,
              24,
              1,
              1
            ],
            //.. Lot's of these
            [
              76,
              6,
              0,
              0,
              0
            ]
          ],
          "expansions": [],
          "summary": {
            "lines": {
              "count": 37,
              "covered": 21,
              "percent": 56
            },
            "functions": {
              "count": 11,
              "covered": 6,
              "percent": 54
            },
            "instantiations": {
              "count": 11,
              "covered": 6,
              "percent": 54
            },
            "regions": {
              "count": 12,
              "covered": 6,
              "notcovered": 6,
              "percent": 50
            }
          }
        },
      }
    }
  ]
}

[f-meloni]

Yes, I had a look at it yesterday, and i think i will support both with two different methods

[f-meloni]

I created the project here https://github.com/f-meloni/danger-swift-coverage/pull/1 Is obviously still WIP, but the files coverage is working :D

[yhkaplan]

Nice! So it looks like you're using xcodebuild's -enableCodeCoverage option and parsing the output with xccov. Should I make a pull request to get it working on Linux with --enable-code-coverage and the file format above?

[f-meloni]

That would be amazing, I wanted to make another method to use the SPM one, and i prepared some of the stuff the actually be re used by the other method too, like the Section struct, but I'm having problems to download the xcode 10.2 beta :/

[f-meloni]

@yhkaplan I've started the implementation for the SPM coverage file, but ATM danger-swift can not be used with swift 5.0 (because Unbox doesn't work with swift 5.0), then i can not add it yet :(