REPL: add syntax highlighting, like the Scala 3 REPL (and Ammonite)

[SethTisue]

as @allanrenucci commented at https://github.com/scala/scala/pull/7645#issuecomment-454558554 ,

Here is the implementation in Dotty for reference. I believe it would be straigforward to port our syntax highlighter

someone want to have a look and see it how even works? does it involve external dependencies?

JLine info page on this is https://github.com/jline/jline3/wiki/Highlighting-and-parsing

[soronpo]

Maybe I'll try to tackle this one.

[SethTisue]

@soronpo are you still interested in working on this?

[soronpo]

Yes, but currently swamped with work to finish my Phd. If anyone wants to tackle this, be my guest.

[SethTisue]

@retronym passed along a link to this which might be useful: https://github.com/scala/scala/commit/60ee9924b7449ec64cffcecd6accd1a856c4fa3a

oh, but Paul ripped some of that out again in scala/scala@317a1056cd8062331964d1bc65f1bfd945538551

[retronym]

scala> val unit = newCompilationUnit("class      C { def foo = for (x <- (1 to 10)) yield null }"); val scanner = new syntaxAnalyzer.UnitScanner(unit); scanner.init(); while (scanner.token != ast.parser.Tokens.EOF) { val x = scanner.toString; val o = scanner.offset; scanner.
nextToken(); println((x, o, scanner.lastOffset))}
('class',0,5)
(id(C),11,12)
('{',13,14)
('def',15,18)
(id(foo),19,22)
('=',23,24)
('for',25,28)
('(',29,30)
(id(x),30,31)
('<-',32,34)
('(',35,36)
(int(1),36,37)
(id(to),38,40)
(int(10),41,43)
(')',43,44)
(')',44,45)
('yield',46,51)
('null',52,56)
('}',57,58)
val unit: $r.intp.global.CompilationUnit = <console>
val scanner: $r.intp.global.syntaxAnalyzer.UnitScanner = eof

[retronym]

If you'd like to detect // comments, a small change to the the parser API would be needed.

Here's what can be done already:

val unit = newCompilationUnit("class      C { def foo = for (x <- (1 to 10)) yield null /* C1 */ // C2}"); val scanner = new global.syntaxAnalyzer.UnitScanner(unit, Nil) { override def processCommentChar(): Unit = { println((s"COMMENT($ch)", charOffset)) } }; scanner.
init(); while (scanner.token != ast.parser.Tokens.EOF) { val x = scanner.toString; val o = scanner.offset; scanner.nextToken(); println((x, o, scanner.lastOffset))}
('class',0,5)
(id(C),11,12)
('{',13,14)
('def',15,18)
(id(foo),19,22)
('=',23,24)
('for',25,28)
('(',29,30)
(id(x),30,31)
('<-',32,34)
('(',35,36)
(int(1),36,37)
(id(to),38,40)
(int(10),41,43)
(')',43,44)
(')',44,45)
('yield',46,51)
(COMMENT( ),60)
(COMMENT(C),61)
(COMMENT(1),62)
(COMMENT( ),63)
(COMMENT(*),64)
(COMMENT(/),65)
('null',52,56)
val unit: $r.intp.global.CompilationUnit = <console>
val scanner: $r.global.syntaxAnalyzer.UnitScanner = eof

[SethTisue]

for the part that involves interfacing with JLine, looking at Dotty's implementation should be helpful. according to https://github.com/lampepfl/dotty/blob/master/docs/blog/_posts/2017-10-16-fourth-dotty-milestone-release.md , “we use code adapted from the Ammonite REPL to provide syntax highlighting”, so maybe look there as well (but note that Ammonite uses Scalaparse which is a dependency we don't want to take on)

relevant source files include:

• https://github.com/lampepfl/dotty/blob/master/compiler/src/dotty/tools/dotc/printing/SyntaxHighlighting.scala
• https://github.com/lampepfl/dotty/blob/master/compiler/test/dotty/tools/dotc/printing/SyntaxHighlightingTests.scala

and files in dotty/tools/repl such as these and perhaps others:

• https://github.com/lampepfl/dotty/blob/master/compiler/src/dotty/tools/repl/JLineTerminal.scala
• https://github.com/lampepfl/dotty/blob/master/compiler/src/dotty/tools/repl/ReplDriver.scala

note that SyntaxHighlighting.scala uses both dotc.parsing.Scanners.Scanner and dotc.parsing.Parsers.Parser but it isn't immediately obvious to me what proportion of each it uses or needs

The key snippets in JLineTerminal are:

    val lineReader = LineReaderBuilder
      ...
      .highlighter(new Highlighter)
      ...
  ...
  private class Highlighter(using Context) extends reader.Highlighter {
    def highlight(reader: LineReader, buffer: String): AttributedString = {
      val highlighted = SyntaxHighlighting.highlight(buffer)
      AttributedString.fromAnsi(highlighted)
    }
  ...

and I think that the use of SyntaxHighlighting in ReplDriver is to highlight definitions the REPL echoes back to the user, so e.g. the second line of this is syntax-highlighted too, not just the input:

scala> def inc(x: Int) = x + 1
def inc(x: Int): Int

[SethTisue]

I have some WIP at https://github.com/SethTisue/scala/commits/repl-highlighting

At present it only uses the scanner, not the parser. No tests yet.

It's a start...

[SethTisue]

How would using the parser look? Rather than talk to the parser ourselves, we should use the presentation compiler, which is designed for this sort of usage, and which the REPL already uses anyway, for doing tab completion. (That happens in ReplCompletion#codeCompletion, which calls intp.presentationCompile(cursor, buf) which returns a PresentationCompileResult giving access to the typed tree.)

Once we have a tree, we can copy the approach taken by SyntaxHighlighting in Dotty, which is to do highlighting in two passes. First it iterates over the tokens returned by the scanner and applies colors (as in the WIP code I've already pushed). Then it walks the trees using traverse and looking for node types of interest such as Ident or ValOrDefDef and applies more colors based on the position information attached to those nodes.

[SethTisue]

As Jason has noted, detecting comments needs special handling. In Dotty, Scanners offers a commentSpans method, backed by commentPosBuf internally and gated by a keepComments flag. I could backport that.

I wondered if there might be something comparable in the presentation compiler already, and, well, there sort of is: ScaladocScanner, which is currently specific to doc comments.