nex icon indicating copy to clipboard operation
nex copied to clipboard

strange lexing behavior

Open databus23 opened this issue 9 years ago • 2 comments

I'm having a hard time understanding the behavior of the lexer in the following case:

/\(/   { fmt.Printf("-> %q\n", yylex.Text()) }
/\)/   { fmt.Printf("-> %q\n", yylex.Text()) }
/[^( ][^ ]*[^ )]/ { fmt.Printf("-> %q\n", yylex.Text()) }
//

package main
import ("fmt")
func main() {
  fmt.Printf("lexing %q:\n", "(rule)")
    NN_FUN(NewLexer(strings.NewReader("(rule)")))
  fmt.Printf("lexing %q:\n", "( rule  )")
    NN_FUN(NewLexer(strings.NewReader("( rule )")))
}

Output of nex -r -s huh.nex:

lexing "(rule)":
-> "("
-> "rule"
lexing "( rule  )":
-> "("
-> "rule"
-> ")"

Why is the lexer swallowing the trailing bracket when there is no space between the content and the surrounding brackets?. This looks like a bug to me. It has something to do with second character class in the content regex ([^ ]*), when I change that to also not match ) it works.

databus23 avatar Oct 16 '15 13:10 databus23

https://github.com/blynn/nex/issues/16 mentions requiring a newline at the end of the file. This looks to apply here too. Adding even a space at the end matched the last bracket correctly "(rule) "

drewwells avatar Oct 17 '15 14:10 drewwells

@drewwells not for me:

lexing "(rule) ":
-> "("
-> "rule"

databus23 avatar Oct 19 '15 08:10 databus23