On fixing expanders

[helins]

This is a place for documenting failing cases related to expanders.

Taken from the docstring of defexpander:

(defexpander expand-once [x e] (e x (fn [x e] x)))

(expand 42 expand-once)

;; Cast error: Can't convert *initial-expander* of class convex.core.lang.Compiler$1 to type interface convex.core.lang.IFn.

Or a variant with a cryptic error:

(defexpander expand-once [x e] (expand x e))

(expand 42 expand-once)

;; Cast error: Can't convert :CAST of class convex.core.data.Keyword to type class convex.core.lang.expanders.AExpander.

[mikera]

@johnmn3 @Engelberg @pbaille you may also be interested in this one.

I am considering:

• Remove the dedicated Expander type and have macros / expanders just be regular functions.
• Have symbol metadata like {:expander true} used in the environment to define what should be used as an expander vs. a regular function (when found in the first position of a list form).
• I believe this is more consistent with how Clojure does macro expansion
• This would simplify the CVM somewhat and reduce risk of error confusing expanders with functions.

Any thoughts?

[pbaille]

I cannot think of a reason to not do so.

[pbaille]

I do not understand this expansion:

I must be missing something here but it seems that it should be this instead :

Why the received expander is not used ? How sub-expressions could be expanded this way ?

[mikera]

OK so I've pushed some changes that convert all expanders to regular functions

expander is no longer necessary (since it would just be the identity function). macro now produces something closer to what @pbaille suggested above.

Still testing, but comments and input welcome!

[pbaille]

I've just done some tests and trying to understand:

;; define an expander that do nothing
;; it should receive the form as-is and stop the expansion (by not using the provided expander)
(defexpander stop [x _] x)


;; here two example of usages, that should (if I understand well) return the given form as-is
(expand '(stop (if true 1 2) (defn id [x] x)) 
         *initial-expander*)

;=> (stop (cond true 1 2) (def id (fn [x] x)))


(expand '(stop (if true 1 2) (defn id [x] x)) 
         (fn [x _] x))
               
;=> (stop (cond true 1 2) (def id (fn [x] x)))
;; apparently the previously suspicious *initial-expander* is not guilty

The stop expander seems to receive 'if and 'defn forms already expanded

but it does not happen with user defined expanders

(defexpander postfix [x e] (e (into () (next (unsyntax x))) e))

(expand '(stop (if true 1 2) (postfix 1 2 +))
         *initial-expander*)

;; => (stop (cond true 1 2) (postfix 1 2 +)) ;; postfix is not expanded

Any ideas ?

[pbaille]

I was suspecting expand to do extra stuff but if I use stop directly it do not change anything

(stop  '(stop (if true 1 2) (defn id [x] x))
        (fn [x _] x))
;;=> (stop (cond true 1 2) (def id (fn [x] x)))

same with:

(expand '(stop (if true 1 2) (defn id [x] x))
         stop)

[pbaille]

I also try to understand how nested quotation works in convex: here some documentation and examples according to the scheme R5rs standard.

see some results (Racket)

(define x 1)
;; some quasiquote examples
`(a b c).         
`(a b ,x)
`(a `(b ,x))
`(a `(b ,',x))
`(a `(b ,,'x))
;; respectively evaluates to
(a b c)
(a b 1)
(a `(b ,x))
(a `(b ,'1))
(a `(b ,x))

in convex we are good for the two firsts (no nesting) but the three others do not match:

It will be a problem to support macros that emits macros and even if we want to deploy a simple macro (we need to quote the code that we are deploying)

[helins]

So you can let go of Expander? Currently a function can be used as an expander but not the opposite:

($.eval/result ($.ctx/create-fake)
               '(expand [1 2]
                        (fn [f e]
                          (e f))))
                          
;; Throws because `e` cannot be converted to a function, but will be resolve once we remove the expander type

[pbaille]

the expander takes two arguments (e f) should be (e f e) I believe. But this is not the source of the error.

[pbaille]

this is working for me:

(expand [1 2]
        (fn [f e]
            (e f *initial-expander*)))

[mikera]

So I think I've managed to replicate the Racket examples (modulo the fact that everything is effectively a quasiquote in Convex at the moment). Following is passing:

@Test 
public void testQuoteCases() {
  Context<?> ctx=step("(def x 1)");
  assertEquals(read("(a b c)"),eval(ctx,"`(a b c)"));
  assertEquals(read("(a b 1)"),eval(ctx,"`(a b ~x)"));
  assertEquals(read("(a '(b ~x))"),eval(ctx,"'(a '(b ~x))"));
  assertEquals(read("(a '(b ~1))"),eval(ctx,"'(a '(b ~~x))"));
  assertEquals(read("(a '(b ~1))"),eval(ctx,"'(a '(b ~~'~x))"));
  assertEquals(read("(a '(b ~x))"),eval(ctx,"'(a '(b ~~'x))"));
}

[helins]

I haven't played enough yet with the new implementation but it looks rather good, doesn't it? @pbaille Did you manage to break something? :p

[pbaille]

not enough yet, but I will try !

[mikera]

I think we are now good, but will leave open a bit longer for any comments or issues.