expand does not return output for macros using quasiquoting

[guberathome]

(expand) works fine using basics like (list):

(defmodule Loop
	(defmacro result [val]
		(list 'set! 'loop-res (list 'Maybe.Just val)) ))
(expand (Loop.result 123))
```		
=> (set! loop-res (Maybe.Just 123))

... but not when  quasi-quoting is involved

```lisp
(defmodule Loop
	(defmacro result [val]
		`(set! loop-res (Maybe.Just %val)) ))
(expand (Loop.result 123))

=> returns no output

Applies to Carp 0.5.0 release version

[scolsen]

This is because calling the macro itself expands the macro, which means the expand call has nothing left to expand. Compare:

鲤 (quasiquote (1 (unquote x) (unquote-splicing y)))
=> (append (cons-last x (cons-last (quote 1) ())) y)
鲤 (defmacro foo [] (quasiquote (1 (unquote x) (unquote-splicing y))))
鲤 (expand foo)
=> (macro foo [] (append (cons-last x (cons-last (quote 1) ())) y))
鲤 (expand (foo))
鲤 ;; nothing emitted

The behavior is the same when using the reader macros.

Probably we should either (or perhaps both):

1. Change (expand (foo)) to just return (foo).
2. Change (expand foo) to return only the expanded body of foo, not its head (drop the macro foo part)

[eriksvedang]

It's pretty standard behaviour to have to quote what you pass to expand, no?

[scolsen]

Maybe. If you quote the examples above they won't actually work, though:

鲤 (expand '(foo))
鲤
鲤 (expand 'foo)
=> foo

The second, expand 'foo seems correct though. The expansion of a quoted symbol is the symbol itself.

The current behavior actually makes a lot of sense to me since macros when called should expand and if you call a macro (like (foo) in (expand (foo)) I'd expect it to expand and for expand to have nothing to expand (since foo was already called).

I think the only thing that's debatable is whether calling expand on a macro name like (expand foo) should only return the body of foo or its full definition (with the body expanded like we do now).

Another point that's debatable is whether or not expansion should always be exhaustive. For instance, common lisp has macroexpand-1 which only expands a form once, whereas our expansion (I think) is always complete, so it'll continue until the body of the macro has no other expandable forms. I do see how something like the limited expansion of macroexpand-1 could be useful.

[guberathome]

Since the Carp macro system seems to be pretty close to CL, it would make things a lot easier to have some functionality like (macroexpand-1). After all my use case was to test, how a user defined macro is expanded.

So how would I go about testing a macro which uses quasiquotes?

[scolsen]

Though, I just realized I'm missing the point here -.-. Whatever behavior we want, it is odd that the use of quasiquotes is not consistent with macros that don't use them.

The only thing I can guess at is that the quasiquote definition forces expansion more eagerly than macros that don't

So how would I go about testing a macro which uses quasiquotes?

For the time being, the best way would be to execute their bodies in the Repl directly or to call expand on the function name, without calling the function, like:

(expand foo)

[scolsen]

For reference, here's the two behaviors when using a non-quasiquoting macro:

鲤 (defmacro bar [] 'goo)
鲤 (expand (bar))
=> goo
鲤 (expand bar)
=> (macro bar [] (quote goo))
鲤 

So what's odd about quasiquoting is:

1. The first case will return nothing at all
2. the second case is the same except the body of the quasiquoted function will be expanded inline in the macro definition.

[guberathome]

Yes, the behaviour should be consistent no matter if quasiquoting is involved or not. With the definition

(defmodule Loop
	(defmacro result [val]
		`(set! loop-res (Maybe.Just %val)) ))
```		

... I get the following in Carp REPL:
```	
> (Loop.result 123)
> (expand '(Loop.result 123) )
>		
```	

There's my problem. ;-)

[scolsen]

I found the general case, I think. This problem crops up when a macro calls another macro (quasiquote is defined as a macro). The reason you get no output for this case is because the fully evaluated form is set! which returns ().

;; (list) is the same as `()` or Unit returned by side-effects like `set!`
鲤 (defmacro foo [] (list))
鲤 (defmacro bar [] (foo))
鲤 (expand bar)
=> (macro bar [] ())
鲤 (expand (bar))
鲤
鲤 (defmacro bar [] (list))
鲤 (expand bar)
=> (macro bar [] (list))
鲤 (expand (bar))
=> ()

I think the reason this happens is that we currently expand macro bodies at definition time:

-- we call this to define macros
dynamicOrMacro :: Context -> Obj -> Ty -> String -> XObj -> XObj -> IO (Context, Either EvalError XObj)                                                     
dynamicOrMacro ctx pat ty name params body = do                                                                                                             
  (ctx', exp) <- macroExpand ctx body  -- Body expanded here!                                                                                                                       
  case exp of                                                                                                                                               
    Right expanded ->                                                                                                                                       
      dynamicOrMacroWith ctx' (\path -> [XObj pat Nothing Nothing, XObj (Sym path Symbol) Nothing Nothing, params, expanded]) ty name body                  
    Left _ -> pure (ctx, exp)  

That's why we see the different behavior with the quasiquote based definition, and you'll see the same effect with any macro that calls another macro.

You can see the behavior in the example, when you call expand bar when bar calls a macro, the body is already expanded in its definition, which means when a call to it is expanded, you don't get (), but rather the evaluation of () which is nothing.

[scolsen]

Unfortunately, simply changing the behavior breaks a lot of stuff.

[scolsen]

Actually, there seems to be a relatively simple fix

[scolsen]

So it's not as simple as I thought. The real problem is that we use the same definition, apply for both macros and functions. apply calls out to eval, which closes the loop and ensures that when we have something like nested macros, they'll be completely expanded and evaluated.

[guberathome]

So what is the way forward?

Anything that might be gained by copying ideas from a CL / Scheme implementation?? (just guessing)

[scolsen]

It's a little bit nuanced, but basically we currently have an evaluator src/Eval.hs, that is responsible for both evaluating and expanding expressions.

Presently, macro expansion basically recurses to a fixpoint and it follows the same logic as function application.

We'll want to create a separate application function for macros that doesn't recurse to the fixpoint—this same function can back an implementation of expand-1.

I can hopefully do this sometime in the next few days.

@hellerve in case he has other insight or thoughts on this.

[hellerve]

I think originally the idea was to separate expansion and evaluation more, so that the top level of the evaluation knows whether the returned thing is the result of expansions or evaluation. Sadly, I haven’t touched the code in a bit, so I’m not sure whether we can avoid this and “just” patch apply to be up for the task of differentiating?