Extend generators

[brandonspark]

What: Currently, generators are only limitedly supported. We don't allow newlines between clauses of the comprehension, so things like

(x for z in [1,2] if true)

parse fine, but if you change it to any of the following:

(c
 for x in [1, 2]
 if true
) 
(c
 for x in [1, 2]
 if true)
(c
 for x in [1, 2] if true)
(c
 for x in [1, 2] if true
)

it breaks.

Why: We should do this, because this actually happens.

How: Added some extra optional terminators. These should allow generous use of newlines and semicolons.

I also needed to add precedence to for_clause and if_clause over for_statement and if_statement. This was so generators would not parse as parenthesized expressions containing for statements, and related.

[savq]

This fails for me on some basic expressions:

# generators.jl
(x for x in xs)

tree-sitter parse generators.jl # master

(source_file [0, 0] - [3, 0]
  (line_comment [0, 0] - [0, 15])
  (generator_expression [1, 0] - [1, 15]
    (identifier [1, 1] - [1, 2])
    (for_clause [1, 3] - [1, 14]
      (for_binding [1, 7] - [1, 14]
        (identifier [1, 7] - [1, 8])
        (identifier [1, 12] - [1, 14])))))

tree-sitter parse generators.jl # brandonspark:extend-generators

(source_file [0, 0] - [3, 0]
  (line_comment [0, 0] - [0, 15])
  (parenthesized_expression [1, 0] - [1, 15]
    (identifier [1, 1] - [1, 2])
    (ERROR [1, 3] - [1, 14]
      (for_clause [1, 3] - [1, 14]
        (for_binding [1, 7] - [1, 14]
          (identifier [1, 7] - [1, 8])
          (identifier [1, 12] - [1, 14]))))))

Works again if you add an if_clause:

# generators.jl
(x for x in xs if x != missing)

tree-sitter parse generators.jl # brandonspark:extend-generators

(source_file [0, 0] - [3, 0]
  (line_comment [0, 0] - [0, 15])
  (generator_expression [1, 0] - [1, 31]
    (identifier [1, 1] - [1, 2])
    (for_clause [1, 3] - [1, 14]
      (for_binding [1, 7] - [1, 14]
        (identifier [1, 7] - [1, 8])
        (identifier [1, 12] - [1, 14])))
    (if_clause [1, 15] - [1, 30]
      (binary_expression [1, 18] - [1, 30]
        (identifier [1, 18] - [1, 19])
        (operator [1, 20] - [1, 22])
        (identifier [1, 23] - [1, 30])))))

We probably need tests without if clauses.

[maxbrunsfeld]

Nice, thanks for catching that @savq.