Investigate adding JUnit format for errors

[bufdev]

[hborham]

Many CI tools have Junit Parsers available to parse output and present it nicely for a build result: TeamCity, Jenkins, and Bitbucket to name a few.

Buf Lint reporting would be easily consumable then by CI's and users if we had an output option for junit xml.

[srikrsna-buf]

This is a draft of the mapping of annotation to jUnit xml can be.

• Starts with a testsuites. All output for a command like lint will be under one testsuites tag.
• Each file will map to a testsuite with name, file attributes mapping to the file path.
• tests and failures will always be equal and map to number of annotations in each file.
• Each type will map to a testcase tag with name attribute filled with type
• Each testcase exactly contains one failure tag containing the error message.

Example:

<testsuites>
    <testsuite name="foo/foo.proto" tests="3" failures="3" errors="0" file="foo/foo.proto">        
        <testcase name="PACKAGE_VERSION_SUFFIX">
            <failure>Package name "foo" should be suffixed with a correctly formed version, such as "foo.v1"</failure>
        </testcase>
        <testcase name="MESSAGE_PASCAL_CASE">
            <failure>Message name "snake_case_message" should be PascalCase, such as "SnakeCaseMessage"</failure>
        </testcase>
        <testcase name="FIELD_LOWER_SNAKE_CASE">
            <failure>Field name "CamelCaseField" should be lower_snake_case, such as "camel_case_field"</failure>
        </testcase>
    </testsuite>
</testsuites>

Alternative simpler approach without file attribute:

<testsuite name="buf-lint" tests="1" failures="3" errors="0">
    <testcase name="foo/foo.proto">
        <failure type="PACKAGE_VERSION_SUFFIX">Package name "foo" should be suffixed with a correctly formed version, such as "foo.v1"</failure>
        <failure type="MESSAGE_PASCAL_CASE">Message name "snake_case_message" should be PascalCase, such as "SnakeCaseMessage"</failure>
        <failure type="FIELD_LOWER_SNAKE_CASE">Field name "CamelCaseField" should be lower_snake_case, such as "camel_case_field"</failure>
    </testcase>
</testsuite>

Both of these lack line and column numbers. Can be added to testsuite tag using properties tag. Not sure if that will be useful.

[tomchwojkofrank]

In the first approach, the second <testcase> should have name="MESSAGE_PASCAL_CASE" right? In the alternative approach, the number of tests seems wrong (it's been a while since I used jUnit). Should it instead be <testsuite name="buf-lint" tests="1" failures="3" errors="0">?

[srikrsna-buf]

@tomchwojkofrank Corrected!