Use SMIE for more structured editing features

[yuyichao]

Ref https://github.com/JuliaLang/julia/issues/9889

Octave-mode uses it, and it offers some nice features:

• When use place point on an "end", the corresponding block beginning (e.g., "if") is highlighted like a matching parenthesis.
• When you do forward-sexp starting at an expression (e.g., if...end), it understands enough to skip the entire expression, like it should, not just the next symbol.
• You can also easily select/delete/copy entire expressions, e.g., whole loops/blocks/function definitions.
• Things like outline/hide-show modes will work better, too, probably.

[tpapp]

I would be happy to contribute to this, is there some work already that one can build on, or should I start from scratch?

[yuyichao]

Nothing that I'm aware of.

[tpapp]

I experimented with SMIE this weekend, and I am not convinced that the operator precedence grammar it provides is a good match for Julia. The problem is that for a lot of constructs, actual parsing is necessary to determine structure, so you end up implementing a dumb parser... in the lexer. For example, consider newlines: white space per se is not significant in Julia, except when it is used to separate expressions unless the parser finds it needs to continue to the next line. Languages close to Julia in syntax (eg Ruby) solve this problem by inserting virtual ; separators between lines. In SMIE, the logic of this goes into the lexer, which makes it really complicated and inelegant, reducing the benefits of SMIE.

I am not pursuing SMIE for Julia for now, but FWIW, some resources if anyone wants to experiment:

1. pages 187–194 of Grune and Jacobs (1990) is a nice introduction to the parsing concepts SMIE uses.
2. many languages already use SMIE, eg see lisp/progmodes in the Emacs source. octave.el and ruby-mode.el document a lot of tricks that would be useful for Julia.
3. AFAIK sml-mode.el was the first mode to use SMIE, and in fact SMIE was written for it. Many of the difficulties mentioned above don't apply to ML, so fewer tricks are needed, making the code not that useful as an example.

[leafpile]

Is there some other way around this?

I was thinking it would be nice to have emacs highlight the matching keywords (say, for and end), which would be available if SMIE is used and one enables show-paren-mode.

[tpapp]

Is there some other way around this?

Someone should revisit SMIE or the cc-mode engine to explore these issues. Help is appreciated.

[non-Jedi]

For what it's worth I've started a direct translation of the lexer from Julia's parser to elisp over at https://github.com/non-Jedi/julia-emacs/blob/smie1/julia-smie.el. I wanted to see how many extra tricks would be needed in the lexer to work with smie. We'll see.

[FelipeLema]

I've been exploring SMIE and the toughest part would be to translate the statefulness of the julia parser.

For example, the use of with-whitespace-newline & without-range-colon would require one to track different :. While this can be done using non-range-: & range-: tokens, that might not be enough because of the combination of using more than one. Would some lexical definitions like non-range-but-can-have-newline-: be enough? Would the definition of forward / reverse tokenization functions be enough?

Maybe I'm being exaggerated, but finding out whether I'm not (crosses fingers) still requires a lot of resources.

[FelipeLema]

I've also been playing with tree-sitter.el, but since the julia parser is far from complete, it cannot be properly used today.

Anyone reading this can contribute to completing the tree-sitter parser, by the way.

Anyone reading this today can help.