Lerche.jl icon indicating copy to clipboard operation
Lerche.jl copied to clipboard

Allow immediate transformation during parsing

Open jamesrhester opened this issue 3 years ago • 0 comments

Building parse trees from very large files (e.g.> 200MB data file) can occupy considerable memory. If the transform is applied immediately at the point that the syntactical construct is recognised, memory can be economised, particularly if only certain information is relevant. This would also allow streaming data. Lark already has this capability via the transform option to Lark, which in Lerche is simply executed following the completion of parsing.

Considerations:

  • If an item for the parse tree is processed by user-supplied functions, the type restraints for nodes of {Tree,Token} are no longer true. Some form of type parameterisation is needed to avoid type instability. This should be under the control of the user.

jamesrhester avatar May 04 '21 04:05 jamesrhester