add to a lexer state dynamically to implement macros?

[mantacid]

I'm using the lexer package to create a lexer for my esolang, and I wanted to implement syntactic/substitution macros by parsing the macro definition into a new rule for a lex-state. Is there a way to accomplish what I want to do?

[mantacid]

I had the idea to use comma-splicing to insert a separate list into the body of the define-lexer macro. This worked, but redefining the list didn't update the function, and I couldn't get it to update no matter how hard I tried. I could really use some advice on how I can avoid creating a new lexer state for each macro that is defined by the user.

[mantacid]

I have just realized that this is the job of the parser, not the lexer. I'd still be interested in learning how that kind of thing would be accomplished, though.

[massung]

Sorry for the late reply. This is something I'd definitely do in the parser. Untested, but the only thing I could think of would be the use of a dynamic variable or subclassing the lexstate to accomplish it in the lexer. For example (again, untested):

(defparameter *macros* (make-hash-table))

(define-lexer macro-lexer (state)
  ("%a+" (pop-lexer state (gethash $1 *macros*))

(define-lexer toplevel-lexer (state)
  ("," (push-lexer state #'macro-lexer :expanded-macro))

But you'd essentially be populating *macros* in an earlier pass akin to the C preprocessor.

To subclass lexstate would require changing other parts of the codebase, but I wonder if that might be a cool feature to add? The idea would be that with-lexer could be supplied an optional :state-class keyword argument (that defaults to lexstate) and you could have an initialize-instance method on that class that sets up any custom state data (like macros) in it. Then you could read/write to the state object within the lexer as it ran?

I'd have to think about this more. What are your thoughts?

[mantacid]

I think it would add a lot of flexibility to the lexer, but I would make the implementation even simpler: store the lexer-function as an unevaluated expression within a closure, which gets read and executed when needed. That way, you can write functions that operate on the closure definition itself to change the lexer's definitions dynamically, while still being able to define an immutable initial state.

I've done something similar with a program I'm working on, where I had a lambda expression literal consed to an ID value. To get the data I wanted, I applied the lambda expression to the value. But if I wanted to change how I operated on the data, I could change the function itself, rather than try to store that new piece of code somewhere else.

[mantacid]

to elaborate on my suggestion for implementation, here is an example of being able to operate on unevaluated expressions:

(defparameter func `(lambda (x) (car x)))
(defmacro construct-thingy (form value)
    `(list ,(eval form) ,value)
)

(defun read-from-thingy (thingy)
    (eval (read-from-string (format nil "(~s `~s)" (car thingy) (cadr thingy))))
)

(defun inverse-operation (thingy)
    (if (eq 'car (car (caddar thingy)))
        (setf (caddar thingy) '(cdr x))
        (setf (caddar thingy) '(car x))
    )
)

(defparameter mything (construct-thingy 'func (cons "foo" "bar")))

(format t "Before: ~a -> ~a~%" mything (read-from-thingy mything))

(inverse-operation mything)

(format t "After: ~a -> ~a~%" mything (read-from-thingy mything))

Using this approach allows you to avoid rewriting the entire code base by providing a closure in which the lexstate is stored and modified by the closure itself.