Unicode test suite fails on Windows

[bmjames]

I think I've seen successful builds on my Windows 7 machine in the past, but I can't pin down when things changed. I've tried a number of things to no avail, including:

• Removing and re-creating the intellij-haskforce project files
• Using IntelliJ IDEA 13 and 14
• Building with JDK 7 and 8
• Re-generating the parser with the latest (pre-release) version (1.2.0.2) of GrammarKit

Currently, when I run the tests, I see these issues every time:

Unicode parsing trouble

The parser fails to recognise Unicode characters as Haskell tokens, even though it seems to be able to read them fine with no file-encoding-related problems. The input files are UTF-8, and the default character set for my Java runtime is UTF-8.

For example, testUnicode00001 in HaskellLexerTest fails because instead of HaskellTokenType.forall ('∀'), the actual parser result for that token is BAD_CHARACTER ('∀'). The same happens for all the other Unicode tokens: BAD_CHARACTER ('→'), et cetera.

Line separators

testCommenter00002 from HaskellFeaturesTest fails with ERROR: Wrong line separators: '...c [] = acc\r\n{-f acc ...' at offset 14 (stack trace). I can see that the test code carefully uses a cross-platform line separator definition. The code responsible for throwing the error (in com.intellij.openapi.util.text.StringUtil) is unfortunately beyond my patience to debug.

File API trouble

HaskellFoldingBuilderTest errors, caused by java.io.IOException: File name cannot contain file path separators: 'tests\gold\features' (stack trace). I assumed this was a problem with IntelliJ's testing libraries, but looking through the code, they seem to use cross-platform file separators correctly. But I must be missing something.

---

If anybody read all that, thanks! I'm posting this in the faint hope that these issues might look familiar to somebody.

[carymrobbins]

Excellent summary @bmjames! It seems that in some cases we actually shouldn't use the platform-specific newline character, and this may also include the file separator. For instance, I resolved some issues on Windows by replacing the platform-specific newline with the standard '\n' character. See commit 88f6017 as an example.

As for the unicode issues, that seems really familiar. I thought I had experienced that due to some localized issue that had been resolved, but maybe not. I have a Windows 7 VM, so I'll try to reproduce these issues and resolve them there.

[carymrobbins]

This may be useful regarding the test suite. Maybe we need to explicitly use the -Dfile.encoding=UTF-8 argument.

http://stackoverflow.com/questions/14823160/java-unicode-issue-on-windows

[bmjames]

Using just \n does seem to fix the newline errors for me; the change is in c0e2f3d3c15f71f77f8 if you'd like to use that.

And thanks for looking at the Unicode issue. I write a fair bit of Haskell on my Windows machine, so it would be great to be able to use HaskForce there.

[bmjames]

Regarding the file.encoding property: If I make the test class print the result of System.getProperty("file.encoding"), then I see UTF-8.

[bmjames]

Your intuition about the path separator was also correct; 9265cbca9b2b contains the changes that made HaskellFoldingBuilderTest work for me.

[carymrobbins]

Ok, the commit above fixes the file separator issue (did that right before I saw your commit, @bmjames). The only thing pending now is the unicode issue, which seems to be related to JFlex. Although, the GhcUtilTest appears to be failing as well, although I'm thinking that it will be pretty trivial to resolve.

@bmjames - Unless you are using UnicodeSyntax, you should be able to build and use the plugin on Windows just fine.

[iostat]

Hey! Just like to add, I've encountered identical Unicode failures from recent master as well on OS X Sierra, 10.12.2 (16C67) and the latest IDEA. I'm running an Ultimate Edition IDEA, if that makes any difference. I've tried adding -Dfile.encoding=UTF-8 to VM Options in the test runner definition, to no avail.

IntelliJ IDEA 2016.3.4
Build #IU-163.12024.16, built on January 31, 2017
JRE: 1.8.0_112-release-408-b6 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o