Parse the error message

[mithro]

It would be good to parse the error message and check that it is what is expected.

[Nic30]

But how to make it compatible with all tools?

[mithro]

@hzeller had ideas. I think he has doing this for https://github.com/zachjs/sv2v/pull/38

[hzeller]

Idea is mostly to send patches to fix the tools. I did that for yosys last year, now currently sv2v, others to follow.

[Nic30]

I have bad feeling that in some cases different tools can recognize different errors and both will be correct at the same time. Also errors in preprocessor may have very different formats. However any standardisation of error messages would be great. I will check the commit in tools time to time so I know what is happening.

[hzeller]

At this first step, we're concerned about the formatting of error messages to have the standard format of

<filename>:<line>:(<column>:)? (Error:|Warning:)? <msg>

This helps that common editors (such as emacs or vi) can easily step through error messages. Some tools have more a 'free-form' version of this format.

Certainly, the tools will report the same problem in different ways or trip over problems in different ways, or have more information to be more verbose about things. So I see less chance to always report exactly the same error - it will probably always somewhat of a difference you'd expect between different tools (think of gcc vs. clang)

With the line-number, we could at least narrow down the place where we expect the problem.

For the longer term of this scope, we should maybe provide some regular expression in the metadata of the tests that determine if error messages contain expected keywords; maybe metadata that can be specified for a particular tool.

[hzeller]

As a follow-up, in the last year I sent a some patches to various tools so that we have a consistent errors/warnings oiutput in the common <filename>:<line>(:col)?: format, which allowed to provide the clickable messages that directly jump to the file in question.

Typically, we expect no error in the tests, so expectation is easy to formulate :)

For the ones that we actually do expect to fail, I think it will be hard to find an expected error with a robust regular expression. But at least we have #449 and #688 that adds a human readable expectation of why a test fails, which comes closest to the spirit of this bug.

I'd say, this bug is ready to be closed ?