Consider capturing bindings by value (FastClosures.jl)

[c42f]

As was briefly mentioned on on Zulip https://julialang.zulipchat.com/#narrow/stream/235161-compiler-frontend/topic/capture-by-value.20closures.3F.20(lowering.20in.20Julia) it might be useful if all _-based closures captured a separate binding.

Alas this would departs in significant but very subtle ways from the lexical scoping of the language which is not great. But on the upside it could be considered because

• _ syntax is for very short expressions and one wouldn't normally want to mutate the bindings captured within
• _ syntax is for calling higher order functions, and in typical uses it would be unusual for the lifetime of the closure to outlive the function it's passed to.
• There's a (small?) chance that the language as a whole will eventually go in this direction.

CC @tkf

[tkf]

BTW, I created https://github.com/tkf/UnderscoreOh.jl which lets you create capture-by-value closures with super ad-hoc syntax like _o.key and _o.x .+ _o.y. I mainly wanted something like Underscores.jl that is recompilation-free and macro-free. But, as a side-effect, it has capture-by-value semantics. It works quite well so far. As you mentioned, it's probably because I'm using it mainly as an argument to higher-order functions. The constraint that I can only create "expressions" (call graphs) with it might also be helping.

Maybe it also helps if you disallow "statements" in @_? ATM, you can do f = let x = 0; @_ begin x = _ + x end; end so changing it to capture-by-value would be breaking. (Although I imagine people wouldn't do this...)

[c42f]

Maybe it also helps if you disallow "statements" in @_

That would be safe but it also seems a little unfortunate given that things like @_ f((y=_^2; y + y^2)) are reasonable enough.

Note that I'm super ok with breaking changes for this package, because there are still several usability problems (#6 / #8) which must be worked out and will be breaking. It appears I was too keen to release version 1.0, but oh well!

Overall the main thing I'm concerned with here is iterating toward the most sensible semantics which could be absorbed into Base in the long term.

[c42f]

I wonder whether there's some Expr head we could have which exposes variable scope in some general way which can make writing these kind of macros more reliable.

It would be fine IMO to disallow

function foo()
    y = 1
    @_ map((y=_^2; y^2), [1,2,3])
    y
end

But the following pure version should be fine

function foo()
    @_ map((y=_^2; y^2), [1,2,3])
end

as should a version where y is explicitly declared local

function foo()
    @_ map((y=_^2; y^2), [1,2,3])
end

We've got Base.@locals (Expr(:locals)) but that exposes this information in a way which can only be used for debugging.

[c42f]

exposes variable scope in some general way

... maybe, it would be occasionally useful to have "macros" which act on desugared ASTs as well as ones which act on the surface syntax.

[tkf]

a little unfortunate given that things like @_ f((y=_^2; y + y^2)) are reasonable enough

Right. I think I agree.

(The hesitancy bit is that I'm still not sure how complex the underscore form anonymous should "allowed" to be.)

exposes variable scope in some general way

... it's like, it would be occasionally useful to have "macros" which act on desugared ASTs as well as ones which act on the surface syntax.

Yes! It'd be super useful for something like FLoops.jl. I guess there is a kind of chicken-and-egg problem since you can't get to the desugared AST without expanding the macro...

I guess it's already possible if you let the macro see the entire function:

@_ function foo()
    map((y=_^2; y^2), [1,2,3])
end

Then, you can analyze the scope using JuliaVariables.jl. Or, it'd be even better if Base.Meta exposes a similar API although it's probably hard unless lowering pass is pulled out from the Scheme code.

For more restricted and composable API, maybe it's nice to have an API @generatewith f expr which is like if @generated but all the processing happens at lowering time and not at compile time (so without type information). Something like @_ would be defined as

macro @_(expr)
    :(@generatewith process_underscores $(esc(expr)))
end

with process_underscores(info, expr) -> expr where info contains scope information etc. But the difficulty is what should happen when the function (info, expr) -> expr changes the scope information. Should it be allowed to change it in the first place? Maybe it's OK to change the scope if it's done depth-first or something?