report lint errors in machine readable format(s)

[toschder]

For further processing it would be very helpful to have an option to get the lint errors, warnings as "machine-readable" format (*.xml, *.json, ....) - like for example JUnit test results.

Detailed description

As an enhancement, getting the lint output in a machine-readable way would be very helpful.

Context

I would post-process the output and render the results to *.html files (for Jenkins) - additionally I´m thinking about reporting the API validation results in SonarQube.

Possible implementation

maybe add another option like --report-as-json, or --report-format=xml|json or whatever.

Your environment

this is environment independend.

[dlouzan]

This would be awesome, I'm also working in a project for internally reporting the compliance of the specs.

[pderaaij]

Any preference towards the output of the report? My idea would be to provide a report in the format of checkstyle. That seems to be a well-known standard and makes integration with CI/CD systems relatively easy.

[elyobo]

junit would be great, also widely accepted; we're generally using jest and eslint with junit output and being able to integrate schema linting in the same way would be great.

[elyobo]

I was able to use the underlying oas-linter and jest to get my junit output by writing a test runner for jest. I didn't use speccy's loader because it doesn't yet support including specs from JS modules, but the change looks to be simple if that could be merged ;)

Not sure whether this approach would work for other test frameworks, the jest integration was pretty straightforward though (the output format could do with some work, not very user friendly, but the junit formatting in using --reporters jest-junit seemed OK.

Pass (one test, that linting was run)

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="jest tests" tests="1" failures="0" time="0.346">
  <testsuite name="undefined" errors="0" failures="0" skipped="0" timestamp="2019-06-07T15:44:08" time="0.092" tests="1">
    <testcase classname=" OAS Linter" name=" OAS Linter" time="0.092">
    </testcase>
  </testsuite>
</testsuites>

Fail (multiple tests, one for each warning from the linter)

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="jest tests" tests="4" failures="4" time="0.39">
  <testsuite name="undefined" errors="0" failures="4" skipped="0" timestamp="2019-06-07T15:50:20" time="0.148" tests="4">
    <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/0 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/0 (schema-properties-type)" time="0.148">
      <failure>schema-properties-type if.then test failed:undefined != object</failure>
    </testcase>
    <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/1 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AddressRegion/oneOf/1 (schema-properties-type)" time="0.148">
      <failure>schema-properties-type if.then test failed:undefined != object</failure>
    </testcase>
    <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/0 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/0 (schema-properties-type)" time="0.148">
      <failure>schema-properties-type if.then test failed:undefined != object</failure>
    </testcase>
    <testcase classname=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/1 (schema-properties-type)" name=" schema-properties-type if.then test failed:undefined != object - #/components/schemas/AuctionListingAddress/oneOf/1 (schema-properties-type)" time="0.148">
      <failure>schema-properties-type if.then test failed:undefined != object</failure>
    </testcase>
  </testsuite>
</testsuites>

[AshwithGundu]

Hi @dlouzan:

Have you worked on report generation of specs.

Example: Let say we are having 10 specs and classifying them into multiple categories like Good, Average, Poor.

Now this 10 specs falls into multiple categories like Good – 4 Average – 3 Poor – 3

If the requirement is not clear please let me know.

Regards, Ashwith

[sandermvanvliet]

I’ve put together a PR that is a step in supporting this (See PR #367), would love to hear some feedback