Unable to generate optimized grammar

[xcoulon]

Hello,

First, thanks a lot for maintaining this project, it's a great library! I'm currently using it in https://github.com/bytesparadise/libasciidoc and it's working really well 🙌

However, since https://github.com/mna/pigeon/commit/9fec3898cef80afe60fbe5df398fceca513566b8 was merged, I've been getting the following error when running the command below in the project's root:

$ pigeon -optimize-grammar -alternate-entrypoints PreparsedDocument,InlineElementsWithoutSubtitution,VerbatimBlock -o ./pkg/parser/asciidoc_parser.go  ./pkg/parser/asciidoc-grammar.peg
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1284d49]

goroutine 1 [running]:
main.main.func1(0x13d3500, 0xc000072020, 0xc0005fde50)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/main.go:87 +0x13d
panic(0x12f2440, 0x15f2fd0)
        /usr/local/Cellar/go/1.12.1/libexec/src/runtime/panic.go:522 +0x1b5
github.com/mna/pigeon/ast.(*grammarOptimizer).optimizeRule(0xc0004bfec0, 0x13d0d80, 0xc0003e2300, 0xc00045d080, 0xc0003e4d20)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_optimize.go:243 +0x309
github.com/mna/pigeon/ast.(*grammarOptimizer).optimize(0xc0004bfec0, 0x13d0d60, 0xc0003e4f00, 0x13d0e40, 0xc0004bfec0)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_optimize.go:182 +0x27ec
github.com/mna/pigeon/ast.(*grammarOptimizer).Visit(0xc0004bfec0, 0x13d0d60, 0xc0003e4f00, 0xc0003e4d20, 0xc0004bfec0)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_optimize.go:39 +0x3e
github.com/mna/pigeon/ast.Walk(0x13d0e40, 0xc0004bfec0, 0x13d0d60, 0xc0003e4f00)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_walk.go:20 +0x55
github.com/mna/pigeon/ast.Walk(0x13d0e40, 0xc0004bfec0, 0x13d0c80, 0xc0004dba90)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_walk.go:41 +0x43b
github.com/mna/pigeon/ast.Optimize(0xc0004dba90, 0xc0001044e0, 0x3, 0x3)
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/ast/ast_optimize.go:464 +0x14e
main.main()
        /Users/xcoulon/code/go/src/github.com/mna/pigeon/main.go:120 +0x106f

The grammar in my project is already quite big: https://github.com/bytesparadise/libasciidoc/blob/master/pkg/parser/asciidoc-grammar.peg and unfortunalely the stack trace does not give much information about the rule(s) that cause the error, so I can't really narrow down the grammar to a simpler form :/

Note: building and running pigeon with the previous commit works like a charm.

[mna]

Hello Xavier,

I'll take a look later today. @breml interestingly, it panics on https://github.com/mna/pigeon/blob/master/ast/ast_optimize.go#L243 where you had left a TODO comment to check for missing references, looks like that's what happens.

Martin

[xcoulon]

Hello Xavier,

I'll take a look later today. @breml interestingly, it panics on https://github.com/mna/pigeon/blob/master/ast/ast_optimize.go#L243 where you had left a TODO comment to check for missing references, looks like that's what happens.

Martin

Thanks for your response, @mna! And yes, I've seen the TODO above the line that caused the panic, but I have no idea how to fix it, sadly :(

[mna]

Took a quick look, it turns out the grammar has the same rule defined twice (FrontMatter), with a reference to a missing rule (YamlFrontMatter). See here: https://github.com/bytesparadise/libasciidoc/blob/master/pkg/parser/asciidoc-grammar.peg#L88-L90

I tried removing the first definition (with the invalid reference) and it builds correctly.

I'll leave the issue open, though, in case Lucas wants to use it to add better error reporting in the optimizer (as hinted in the todo comment).

Martin

[xcoulon]

Took a quick look, it turns out the grammar has the same rule defined twice (FrontMatter), with a reference to a missing rule (YamlFrontMatter). See here: https://github.com/bytesparadise/libasciidoc/blob/master/pkg/parser/asciidoc-grammar.peg#L88-L90

I tried removing the first definition (with the invalid reference) and it builds correctly.

I'll leave the issue open, though, in case Lucas wants to use it to add better error reporting in the optimizer (as hinted in the todo comment).

Martin

Thank you so much, @mna 🙌 It actually works much better now, after renaming the rule on line 90 to YamlFrontMatter. And yes, I agree with you: better error reporting would be very helpful in this case 🙂