Syntax highlighting lexer error

[rodlogic]

(separate issue created from #67)

I got an exception while loading the https://github.com/nominolo/lambdachine project in HaskForce:

[ 931460]  ERROR - napi.project.CacheUpdateRunner - Error while indexing /opt/lambdachine/tests/Bc/Set1.hs
To reindex this file IDEA has to be restarted 
java.lang.Error: com.haskforce.highlighting._HaskellSyntaxHighlightingLexer: Error: could not match input
    at com.haskforce.highlighting._HaskellSyntaxHighlightingLexer.zzScanError(_HaskellSyntaxHighlightingLexer.java:706)
    at com.haskforce.highlighting._HaskellSyntaxHighlightingLexer.advance(_HaskellSyntaxHighlightingLexer.java:1240)
    at com.intellij.lexer.FlexAdapter.locateToken(FlexAdapter.java:95)
    at com.intellij.lexer.FlexAdapter.advance(FlexAdapter.java:76)
    at com.intellij.lang.cacheBuilder.DefaultWordsScanner.processWords(DefaultWordsScanner.java:88)
    at com.intellij.psi.impl.cache.impl.id.IdTableBuilding$WordsScannerFileTypeIdIndexerAdapter.map(IdTableBuilding.java:139)
    at com.intellij.psi.impl.cache.impl.id.IdTableBuilding$WordsScannerFileTypeIdIndexerAdapter.map(IdTableBuilding.java:126)
    at com.intellij.psi.impl.cache.impl.id.IdIndex$4.map(IdIndex.java:85)
    at com.intellij.psi.impl.cache.impl.id.IdIndex$4.map(IdIndex.java:79)
    at com.intellij.util.indexing.MapReduceIndex.update(MapReduceIndex.java:398)
    at com.intellij.util.indexing.FileBasedIndexImpl.updateSingleIndex(FileBasedIndexImpl.java:1758)
    at com.intellij.util.indexing.FileBasedIndexImpl.doIndexFileContent(FileBasedIndexImpl.java:1690)
    at com.intellij.util.indexing.FileBasedIndexImpl.indexFileContent(FileBasedIndexImpl.java:1638)
    at com.intellij.util.indexing.UnindexedFilesUpdater$2.consume(UnindexedFilesUpdater.java:101)
    at com.intellij.util.indexing.UnindexedFilesUpdater$2.consume(UnindexedFilesUpdater.java:97)
    at com.intellij.openapi.project.CacheUpdateRunner$MyRunnable$1.run(CacheUpdateRunner.java:279)
    at com.intellij.openapi.application.impl.ApplicationImpl.runReadAction(ApplicationImpl.java:905)
    at com.intellij.openapi.project.CacheUpdateRunner$MyRunnable$2.run(CacheUpdateRunner.java:298)
    at com.intellij.openapi.progress.impl.ProgressManagerImpl$3.run(ProgressManagerImpl.java:194)
    at com.intellij.openapi.progress.impl.ProgressManagerImpl.registerIndicatorAndRun(ProgressManagerImpl.java:281)

There are many files that fail to lex such as compiler/Lambdachine/Interp/Exec.hs, tests/Bc/Bc0007.hs, tests/Bench/Fibon/Agum/UnificationMatching.hs, tests/Bench/Nofib/Spectral/Boyer.hs, tests/Bench/Nofib/Spectral/Lambda.hs, tests/Bc/Jit0008.hs, Bc/Compile2.hs.

[carymrobbins]

@rodlogic - Does the editor still go haywire on you? Unfortunately, I haven't been able to reproduce that particular symptom; however, the lexer error definitely appears. It seems like it only occurs on the first lex of the file, so it may have something to do with the internal state of the lexer.

[carymrobbins]

Technical note - You can use the %debug option to have the lexer spit out debugging info to the console. See the JFlex manual for more info -

http://jflex.de/manual.html#Standalone

[rodlogic]

@carymrobbins I have been a bit thin on time on my end. However, I did try including %debug in the .flex file, but I can't find the main method after a recompile. Is there anything else that needs to be done?

[carymrobbins]

@rodlogic - It seems that maybe the IntelliJ-forked version of JFlex doesn't generate this main function, which makes sense since IntelliJ hooks into it differently (I was looking for the main function at first, too).

Instead, when you have the %debug option enabled (and you re-generate the lexer) you can run the plugin, either by running the entire plugin in run or debug mode or running a test case.

The odd part is that running some of those files individually through the lexer doesn't seem to produce the error, so my hunch is that we have something stateful that's breaking.

[carymrobbins]

Here's another stack trace which seems a little different -

Error while indexing /home/crobbins/projects/intellij-haskforce/tests/gold/parser/Hello00003.hs
To reindex this file IDEA has to be restarted
java.lang.IndexOutOfBoundsException
    at java.nio.Buffer.checkIndex(Buffer.java:546)
    at java.nio.CharBuffer.charAt(CharBuffer.java:1278)
    at com.haskforce.highlighting._HaskellSyntaxHighlightingLexer.advance(_HaskellSyntaxHighlightingLexer.java:813)
    at com.intellij.lexer.FlexAdapter.locateToken(FlexAdapter.java:95)
    at com.intellij.lexer.FlexAdapter.getTokenType(FlexAdapter.java:58)
    at com.intellij.lang.cacheBuilder.DefaultWordsScanner.processWords(DefaultWordsScanner.java:74)
    at com.intellij.psi.impl.cache.impl.id.IdTableBuilding$WordsScannerFileTypeIdIndexerAdapter.map(IdTableBuilding.java:119)
    at com.intellij.psi.impl.cache.impl.id.IdTableBuilding$WordsScannerFileTypeIdIndexerAdapter.map(IdTableBuilding.java:106)
    at com.intellij.psi.impl.cache.impl.id.IdIndex$4.map(IdIndex.java:85)
    at com.intellij.psi.impl.cache.impl.id.IdIndex$4.map(IdIndex.java:79)
    at com.intellij.util.indexing.MapReduceIndex.update(MapReduceIndex.java:398)
    at com.intellij.util.indexing.FileBasedIndexImpl.updateSingleIndex(FileBasedIndexImpl.java:1831)
    at com.intellij.util.indexing.FileBasedIndexImpl$23.run(FileBasedIndexImpl.java:1760)
    at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.freezeFileTypeTemporarilyIn(FileTypeManagerImpl.java:510)
    at com.intellij.util.indexing.FileBasedIndexImpl.a(FileBasedIndexImpl.java:1718)
    at com.intellij.util.indexing.FileBasedIndexImpl.indexFileContent(FileBasedIndexImpl.java:1703)
    at com.intellij.util.indexing.UnindexedFilesUpdater$2.consume(UnindexedFilesUpdater.java:108)
    at com.intellij.util.indexing.UnindexedFilesUpdater$2.consume(UnindexedFilesUpdater.java:105)
    at com.intellij.openapi.project.CacheUpdateRunner$MyRunnable$1.run(CacheUpdateRunner.java:226)
    at com.intellij.openapi.application.impl.ApplicationImpl.tryRunReadAction(ApplicationImpl.java:1178)
    at com.intellij.openapi.project.CacheUpdateRunner$MyRunnable$2.run(CacheUpdateRunner.java:246)
    at com.intellij.openapi.progress.impl.CoreProgressManager$2.run(CoreProgressManager.java:142)
    at com.intellij.openapi.progress.impl.CoreProgressManager.a(CoreProgressManager.java:446)
    at com.intellij.openapi.progress.impl.CoreProgressManager.a(CoreProgressManager.java:443)
    at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:392)
    at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:54)
    at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:127)
    at com.intellij.openapi.project.CacheUpdateRunner$MyRunnable.run(CacheUpdateRunner.java:240)
    at com.intellij.openapi.application.impl.ApplicationImpl$8.run(ApplicationImpl.java:369)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

[carymrobbins]

I just uncovered something while looking at an issue on YouTrack -

It sounds like there's something wrong with the lexer. There are some subtleties with it, not obvious at all. For example, it shouldn't return 0 from getState in some position unless it really can be restarted from that position. It might be not thread-safe. It might also not clear its state thoroughly when being restarted.

This is actually a statement in regards to a completely different kind of issue, but it might actually apply here. If we are calling yybegin(YYINITIAL) (which is state 0) and we have modified the lexer's internal state (like stateStack or qqLevel) a new lexer instance may not be able to pick up from exactly that position.

Also, looking at the _HaskellSyntaxHighlightingLexer.flex, we can probably simplify it even further and remove state altogether, relying on the HaskellAnnotator for any special, parse-tree specific highlighting cases.