Output format suited for GitHub job summaries

[ParanoidUser]

Is your feature request related to a problem? Please describe. Last year, GitHub introduced job summaries to output custom Markdown content related to the job run. We're trying to make use of this feature to add clarity to our workflow, but it seems like none of the supported formats are ready to produce output suited for the summary page.

Describe the solution you'd like It would be nice to have a special format (in plain text or Markdown) to generate output better suited for the GitHub summary page. While the stats command is the best candidate to generate summary content, this format might be useful for other commands as well (e.g. lint warnings that don't fail the job).

Describe alternatives you've considered Execute command to generate output, redirect standard streams to GITHUB_STEP_SUMMARY env.

redocly stats public/codewars.yaml >> $GITHUB_STEP_SUMMARY 2>&1

Additional context Here is how the output is displayed in the job summary:

[31m🚗 References: 30 
[39m[35m📦 External Documents: 0 
[39m[37m📈 Schemas: 14 
[39m[33m👉 Parameters: 3 
[39m[36m🔗 Links: 0 
[39m[32m➡️ Path Items: 4 
[39m[33m👷 Operations: 4 
[39m[37m🔖 Tags: 2 
[39m[90m
public/codewars.yaml: stats processed in 14ms

[39m

Credits Kudos to the whole Redocly team for giving our community such a great library! 🥳

[adamaltman]

💡 idea: maybe add a --format=markdown option?

[adamaltman]

💡 idea: use GitHub annotations to annotate specific issues on the actual lines they occur?

[adamaltman]

FYI @ParanoidUser the lint command has a summary format and it seems like a good candidate for the job summary.

Sample output:

➜  spec redocly lint --format=summary
validating /openapi.yaml...
error   operation-description assertion: 12
warning operation-4xx-response: 22
warning info-license: 1

/openapi.yaml: validated in 374ms

[ParanoidUser]

Thanks for pointing that out! The idea with stats was to see a quick summary of what the bundle phase results in. Looking back on the idea of adding --format=markdown, it should be called --format=summary. This would be more in line with current conventions.

Off topic: Is there a way to make the lint process fail when there are warnings in the spec? I discovered the --lint-config option which sets the severity level for the config file. It would be useful to have the same capability but for API definitions.

[adamaltman]

Every rule can be configured individually. Are you purely using the minimal or recommended rules with no tweaks? (Would love to learn more about how people use it.)

Here is an example of extensive custom rules and changing of severity levels too:

https://github.com/Rebilly/api-definitions/blob/main/redocly.yaml#L22

[ParanoidUser]

We're currently using the recommended set of rules with no changes. Our goal is to provide bundled & tested spec that follows all the best practices downstream to generate SDKs for different platforms and languages. To achieve higher compatibility and quality standards we decided to follow zero-warning/error policies. Meaning that even non-critical issues with valid syntax need to be fixed before publishing the spec.

We assume the Redocly team has the best vision of what is good & bad in the OAS field and we want to leverage this knowledge. Creating a custom config with bumped severity levels for all warnings to errors might be the way of doing this. But, eventually, it will become outdated and would require periodic sync-ups with the latest practices to keep up. That is what we're trying to avoid.

Ideally, we would like to continue using all recommended rules as is, but with tweaked severity to fail tests even on warnings.

[tatomyr]

Hi @ParanoidUser,

using all recommended rules as is, but with tweaked severity to fail tests even on warnings.

Is it related to this issue?

[ParanoidUser]

Bingo! Thanks for sharing, @tatomyr! Yes, it looks like the use cases are identical.

[tatomyr]

A similar request: https://github.com/Redocly/redocly-cli/issues/1304

[lornajane]

Use the stats command with --format=markdown to get the github job summary you requested. (see also the docs: https://redocly.com/docs/cli/commands/stats/#markdown

To address a couple of other things in this thread before I close it:

• @ParanoidUser we added something that is the recommended ruleset but where everything is an error: it is recommended-strict. Check out the list of rulesets for more information https://redocly.com/docs/cli/rules/#rulesets.
• The lint command can now annotate, use --format=github-actions when you use the command in a github action. Docs and an example: https://redocly.com/docs/cli/commands/lint/#github-actions

If anyone needs anything else, let's get a new issue open for each thing. Thanks everyone :)