Design thoughts on the new parser

[nocturn9x]

Since we'll be rewriting the whole JAPL backend very soon, I'd like this time to be the last one that such a big change is needed. To achieve this, we need a scalable and modular backend that can be easily extended in the future, which is what lead me to this design proposal for the JAPL parser (which, I remember, will be a top-down recursive-descent parser).

I'll make my example by taking a piece of code from japl-python's parser:

def unary(self):
    """Parses unary expressions"""

    if self.match(TokenType.NEG, TokenType.MINUS):
        operator: Token = self.previous()
        right: Expression = self.unary()
        return Unary(operator, right)
    return self.call()

This is the usual design for a recursive-descent top-down parser: it tries to parse something at a given precedence level (unary expressions in this case) and if it fails to do so, it scales up to the higher precedence level (function calls in our example). This is true for all the precedence levels.

The proposed design would be to have a standard interface to add a given "handler" to the parser and bind it to a precedence level. The handler should try to parse an expression (or a declaration/statement) at its own precedence level and resort to calling the handler that's one level higher than itself in the precedence table until the first handler is reached and an exception is thrown if even that one fails to parse. We could devise some procedures in the fashion of addHandler(parser, precedence, procedure) where a given parser objects gets added a procedure that parses an expression at the given precedence level. The procedure must take a single positional argument, which is the parser object itself. Handlers could be stored in a hashmap where the precedence level is the key (either an integer or an enum entry emulating one) and the value is a list of procedures: this is needed to allow multiple handlers at the same precedence level (for example, floor division and modulo division share the same precedence). Handlers must still be called from the handlers themselves, maybe using a wrapper such as getNextHandler(parser, precedence). This procedure will return either the next procedure in the same precedence group, or if there are no other procedures at the current precedence level, return the first one from the list of the next precedence level. If getNextHandler is called with the lowest precedence, it will raise a syntax error.

[nocturn9x]

Update: This is actually a lot more trickier than it sounds. I'll leave the issue open, but currently my focus is to get up and running ASAP with JAPL v2 and this is frankly not a priority anymore