Improve output readability: warnings vs errors

[etanshaul]

Hello! we are successfully using this verifier on our plugin's repo. We have it configured currently so that only compatibility_problems stop the build. We do unfortunately have a lot of deprecation warnings as well that we intend to fix at some point. However, it is currently quite hard to parse the output of the check to get at the actual blocking issue.

Request:

Format the verification output in such a way that makes it easier to get out the violations of the configured failure_levels. I'm leaving it open as to how that would look, but currently we have to scroll through a lot of logs / make extensive use of ctrl+f.

[ChrisCarini]

Hi @etanshaul - this request makes sense at first reading. Do you have any GH workflow execution(s) you are able to share that I could look at to get a better sense of the painpoint here?

[etanshaul]

hey @ChrisCarini - unfortunately I can't share the full execution, but hopefully these snippets help get the idea across:

When I open the check after noticing the failed run on my PR, I see:

This is the first piece of friction. My eyes are drawn to the error lines 1104 - 1111, but these doesn't provide anything useful / actionable.

After Some digging, I realize I need to expand the line that says:

Running verification on /github/workspace/***-*.zip for /github/home/ides/ideaIC-2020.2 /github/home/ides/ideaIU-2020.2 /github/home/ides/ideaIC-LATEST-EAP-SNAPSHOT /github/home/ides/ideaIU-LATEST-EAP-SNAPSHOT...

In here, there are hundreds of lines including:

Standard things like verificaiton setup:

...
Starting the IntelliJ Plugin Verifier 1.256
  Verification reports directory: verification-2021-05-05 at 21.54.24
  2021-05-05T21:54:25 [main] INFO  verification - Reading IDE /github/home/ides/ideaIC-2020.2
  2021-05-05T21:54:25 [main] INFO  c.j.p.options.OptionsParser - Preparing IDE /github/home/ides/ideaIC-2020.2
  Using Java from JAVA_HOME: /usr/lib/jvm/java-1.8-openjdk
...

To deprecation warnings like:

Deprecated API usages (174): 
      #Deprecated class com.intellij.openapi.fileTypes.StdFileTypes reference
          Deprecated class com.intellij.openapi.fileTypes.StdFileTypes is referenced in 

As mentioned, these deprecation warnings are currently not blocking PRs as we have configured them.

Finally, after sifting through the logs, I find something like this:

Plugin com.google.gct.core:21.4.1-SNAPSHOT against IC-211.7142.45: 1 compatibility problem, some of which may be caused by absence of optional dependencies in the target IDE IC-211.7142.45. 21 usages of scheduled for removal API ...  Missing dependencies: 

And slightly further below:

Compatibility problems (1): 

With the "real" issue.

There is no easy way to arrive at the above without sifting through all the logs or searching through matches of the string "compatibility problems".

Hopefully that helps!

[ChrisCarini]

@all-contributors please add @etanshaul for ideas

[allcontributors[bot]]

@ChrisCarini

I've put up a pull request to add @etanshaul! :tada: