Negative test (-compile, or expectation of compilation error)

[eed3si9n]

Something that came up recently in places like

• https://github.com/sbt/zinc/pull/370
• https://github.com/sbt/zinc/pull/424

is the effectiveness of -compile.

problem

I personally think it's important that we test non-compilation, but I do admit that blankly accepting -compile is a bit dangerous because things could fail on all sorts of other reasons (like Windows).

idea

Perhaps what we need is to extend scripted to check for a specific error message, similar to the negative test done in scala/scala (see https://github.com/scala/scala/tree/v2.12.3/test/files/neg, for example https://github.com/scala/scala/blob/v2.12.3/test/files/neg/abstract-class-error.check)

Our syntax could be:

> -compile [abstract-class-error.check]

We then will check in a file named abstract-class-error.check and compare the error message upon failure.

[eed3si9n]

/cc @jvican

[jvican]

I love this. This is what I proposed in our previous meeting when I said "being more similar to partest and check for concrete errors". I'm happy to implement this.

[jvican]

By the way, in the scripted handlers we already have checkWarning and checkError which get all the compiler output from the analysis file and checks that certain errors have indeed been reported. There's only one scripted test using checkWarning.

What should be done is merge the checking step with the compilation step and put everything in a check file. I prefer it this way since it's more idiomatic and allow us to check the whole error message, not just the title.