RFC: curry underscore arguments to create anonymous functions

[stevengj]

This PR addresses #554, #5571, and #22710 by "currying" underscores in function calls like f(_,y) into anonymous function expressions x -> f(x,y). (Note that _.foo works and turns into x -> x.foo since it is equivalent to a getfield call, and _[i] works and turns into x -> x[i], since it is equivalent to a getindex(_,i) call.)

This will help us get rid of functions like equalto (#23812) or occursin (#24967), is useful for "destructuring" as discussed in #22710, and should generally be convenient in lots of cases to avoid having to explicitly do x -> f(x,y).

Some simplifying design decisions that I made:

• The currying is "tight", i.e. it only converts the immediately surrounding function call into a lambda, as suggested by @JeffBezanson (and as in Scala). So, e.g. f(g(_,y)) is equivalent to f(x -> g(x,y)). (Note that something like find(!(_ in c), y) will work fine, because the ! operator works on functions; you can also use _ ∉ c.) Any other rule seems hard to make comprehensible and consistent.

• ~~Only a single underscore is allowed. f(_,_) throws an error: this case seems ambiguous to me (do you want x -> f(x,x) or x,y -> f(x,y)?), so it seemed better to punt on this for now. We can always add a meaning for multiple underscores later.~~ Similar to Scala, multiple underscores are converted into multiple arguments in the order they appear. e.g. f(_,y,_) is equivalent to (x,z) -> f(x,y,z). See rationale below.

The implementation is pretty trivial. If people are in favor, I will add

• [x] Tests
• [x] Documentation
• [ ] Fix interaction with broadcasting f.(x, _)
• [ ] Fix interaction with keyword arguments f(x; y=_)
• [ ] Support chained comparisons, e.g. 3 ≤ _ ≤ 10, since they parse as a single expression?

[stevengj]

Note that if this is merged, using an underscore as an rvalue should probably become an error (currently it is deprecated). As an lvalue, it is fine — we can keep using it for "discarded value" (#9343).

[StefanKarpinski]

My gut feeling is that I'd rather not rush this and that a few ad hoc legacy currying functions in Base aren't going to kill us. Although I have to say that the simplicity of the rule has the right feel to me.

[stevengj]

I agree that we don't want to rush new features like this, but I feel like this idea has been bouncing around for a long time (since 2015), the reception has been increasingly positive, and we keep running into cases where it would help as we transition to a more functional style (thanks to fast higher-order functions).

[stevengj]

(I guess technically this is partial function application, not currying. Note that Scala does something very similar with underscores, and it allows multiple underscores. In a some circumstances Scala apparently requires you to explicitly declare the type of the underscore argument, though, at least if you want its type inference to work.)

[yurivish]

What potential backwards incompatibilities does this rule expose us to?

I know Stefan spent a while trying to find a simple set of rules for determining the "tightness" of a partial application expression and found there were some difficulties. Merging this would close the door on changing the tightness rule in 1.0.

Is there anything else?

[stevengj]

@yurivish, using _ as an rvalue is already deprecated. So this should be backward-compatible with non-deprecated code.

[stevengj]

Scala's rule for "tightness" is (Scala Language Specification, version 2.11, section 6.23.1):

An expression e of syntactic category Expr binds an underscore section u, if the following two conditions hold: (1) e properly contains u, and (2) there is no other expression of syntactic category Expr which is properly contained in e and which itself properly contains u.

which seems essentially the same as the one I've used here (i.e. the innermost expression that is not itself an underscore "binds" it). Scala is more general in two ways:

• Scala allows underscores to appear multiple times to denote multiple arguments, e.g. f(_,_,z) would be x,y -> f(x,y,z).
• Scala allows underscores to appear in more than just function calls. e.g. you can do if _ foo; else bar; end

Both of these could easily be added later, after my PR, since they are a superset of my functionality.

Regarding Stefan's rules, I found them pretty complicated and confusing: why should 2_+3 produce x -> 2x+3, but sqrt(_)+3 produces (x->sqrt(x))+3? Anyway, as the expressions get more complicated than a single function call, it becomes less onerous to simply type x->.

[yurivish]

@stevengj Of course, you're right – I think I misused the phrase "backwards incompatibility". I meant to ask whether there were any "terse partial function application" syntaxes that we may want to introduce in the future that would conflict with the functionality implemented in this branch.

If it turns out that future enhancements would almost certainly be supersets of this one, then, well, fantastic. 😄 I'd personally love to see something like this in the language so long as it's not limiting our options too much down the line.

Edit, written before the last paragraph in the preceding post: Hmm. Expressions with operators (like 3 * _ + 2) would be turned into (x -> 3 * x) + 2 in most cases, unless they happen to be lowered to a single "call" (like 1 + _ + 3).

[ararslan]

I don't have strong feelings one way or another about this change but I've played around with it a little and I noticed something kind of odd:

julia> _ + 1
#1 (generic function with 1 method)

julia> _

WARNING: deprecated syntax "underscores as an rvalue".
ERROR: UndefVarError: _ not defined

That is, the behavior at the top level is surprising. I kind of wonder whether it might be better to adjust the parsing behavior in that context, kind of like how generators require parentheses at the top level:

julia> i for i in 1:10
ERROR: syntax: extra token "for" after end of expression

julia> (i for i in 1:10)
Base.Generator{UnitRange{Int64},getfield(, Symbol("##3#4"))}(getfield(, Symbol("##3#4"))(), 1:10)

Also, I realize this is expected based on the stated rules, but it took me by surprise:

julia> map(_, [1,2,3])
#9 (generic function with 1 method)

I'm not entirely sure what I expected with that expression, but it wasn't that. 😛

Behavior aside, I love how tiny the implementation actually is! It's impressively minimal for a potentially powerful feature.

[andyferris]

Really cool :)

@ararslan The behavior you highlight as odd seems natural to me.

(An aside: what is an unbracketed Generator ambiguous with, or stated another way, why does it require brackets?)

[piever]

Really nice, also for the data ecosystem where the frequently used function i->i.a can now be written as _.a (see #22710, example application mean(_.a, df) where df is an iterable of NamedTuples).

[rfourquet]

As |> is being deprecated, with the idea to have it dovetail nicely with some syntax for currying functions in the future, it would be nice to be sure that this PR's change will be compatibe with the overall design. I think I would prefer to have the whole thing fleshed out before introducing the feature, which can be introduced in 1.x, and as Stefan said few "ad hoc legacy currying functions in Base" is fine in the meantime.

[stevengj]

@ararslan, we could certainly implement (_) as a shorthand for the identity function, but I'm not sure it's worthwhile. It seems better to leave _ as an r-value (i.e. not as an argument to a function call) deprecated/disallowed. We can always add a meaning later.

I would rather not require parens around e.g. _ ≤ 1 and _.a, which are unambiguous and nicely terse already.

[stevengj]

@rfourquet, this is totally orthogonal to the piping syntax (#20331); I'm not sure you think why the latter would affect this.

We've been discussing a currying/partial-application shorthand for literally years now, and all of the discussions seem to have been converging towards underscore syntax, which has a long track record in Scala.

[piever]

The underscore syntax is actually already used, for example in in the Query package, via macros (see docs). The only difference with Query is that there there is no tight binding, for example @filter(df, _.a > _.b) would actually correspond to filter(i->i.a > i.b, df) rather than filter((i->i.a) > (i->i.b), df). It'd be really cool to also make this "loose binding" possible without macros but I'm really not sure how. There was one interesting related idea by @stevengj here to use double underscore for loose binding.

[rfourquet]

this is totally orthogonal to the piping syntax (#20331); I'm not sure you think why the latter would affect this.

This link concerns a removal of the piping syntax without plan to re-introduce it later, which was discussed recently at #5571. It's true that curying syntax can exist independantly to piping syntax, but the design of piping syntax, which deals directly with curryed functions, could influence how the currying syntax should be designed.

[stevengj]

I think it would be short-sighted to tie currying syntax to piping syntax. Currying (partial function application) is useful for lots of things beside piping.

[stevengj]

I could imagine a "loose-binding" syntax like _(...), e.g. _(_.a > _.b). This would still be pretty terse, would be unambiguous and not restricted to certain precedence levels, is easy to implement, and would be compatible with this PR.

[bramtayl]

Is _(_.a > _.b) that much shorter than _ -> _.a > _.b ?

[stevengj]

@bramtayl, possibly not. In general, I'm skeptical of the need for this kind of terse syntax beyond single function calls. And all the attempts to come up with a "loose binding" DWIM syntax seems to lead to rules that are very confusing and context-dependent.

[stevengj]

@yurivish, in this PR, _ only works in the arguments of a function. _(arg1, arg2) is currently not allowed. It could be added later, of course.

[yurivish]

@stevengj I deleted my comment right after posting when I realized what I said didn't make sense .(my example was map(_(arg1, arg2), list_of_functions)).

But I thought it didn't make sense for a different reason — because it would turn the entire expression into an anonymous function. It seems your approach is even more conservative than I realized. 😄

Would map(_[3], arrays) work, since it desugars to a getindex call where the _ is in argument position?

[stevengj]

@yurivish, yes map(_[3], arrays) works as-is in this PR.

[yurivish]

I have to say that the more I think about this proposal the more I like it.

+💯 from me.

[andyferris]

I think it would be short-sighted to tie currying syntax to piping syntax.

It's possible that the reason piping has been raised is that there was a proposal (e.g. #24886 plus discussions on discourse I think) to have _ do currying with a scope boundaries defined by |>, so that we can have expressions like data |> f(_, 2) |> g(_...) |> _[1] * h(_, i(_)), short for data |> x -> f(x, 2) |> x -> g(x...) |> x -> x[1] * h(x, i(x)).

In this case |> provides explicitly the tightness of the binding of _. Now that _ as rhv is deprecated and #24886 deprecates |> as a standard infix operator (mapping to a function), the possibility of implementing piping-bound currying has arisen. In fact, I think the main argument of #24886 was in-fact to allow it to come back as a more complicated and complete beast.

I was a bit torn on this one, because I think piping should be supported and currying / anonymous functions are heavily used in piping and so I feel a convenient pipe-scoped currying is warranted (the visual similarity of |> and -> alone make it hard to decode piped expressions). However, this PR is very convenient in a range of other situations.

Still, I wouldn't say it's short-sighted to consider piping here. I will note that this tight-binding approach still gives us good readability for many common cases for data piping operations like:

data |> map(f, _) |> filter(g, _) |> innerjoin(_, some_other_data) |> reduce(h, v0, _)

so in practice -> may only be required in a minority of cases.

[stevengj]

@andyferris, I think that if the binding of _ were delimited by |>, it would be pretty unfortunate:

• Currying is extremely useful outside of |>. Either you don't allow _ currying without |> (which is annoyingly limited), or you give it a completely different meaning in non-|> contexts (which is confusing).
• In general, I think we should be very wary of syntax where the meaning of some Expr(...) depends on an enclosing expression. There are cases where it is appropriate (e.g. x::T can be a declaration or an assertion, depending on context), but the Julia style has been to "pun" with care.

[stevengj]

("Loose" binding for a selected set of operators like + could also be achieved on top of this PR by defining function-composition methods for them, analogous to what we did for ! in #17155. But I doubt defining lots of function + number and similar methods one by one is a good idea.)

[stevengj]

Grepping the Julia source code for \w+ *->, I'm finding what appears to be 100+ cases (at least) where this syntax could be used (i.e. single-argument anonymous functions that pass their argument to a single function call).

[andyferris]

@stevengj Agreed - I personally feel that overall this PR is much better than that idea (I did however think it was worth mentioning the earlier discussions/thoughts on piping, for the record).

[StefanKarpinski]

"Loose" binding for a selected set of operators like + could also be achieved on top of this PR by defining function-composition methods for them

This is a really interesting observation since cases like 2_ + 1 or 2_ + 3_ were some of the motivations for looser binding of _. If we were to do this, the guideline would seem to be that operations that have non-function-call syntax would be worth considering. Unfortunately, that's a pretty large set of things in Julia, so I'm not sure where that leaves us.

[stevengj]

Wow, great to see a green CI again.

[stevengj]

@StefanKarpinski, I don't think supporting "loose" semantics for 2_+3 and not sin(_)+3 is especially worthwhile; I think it would just be confusing. That's also why I don't think we'd want to do it one-by-one for a whitelist of operators. The only way to make such "loose" binding work well, in my opinion, would be to have some kind of automatic function-composition fallback instead of MethodError when a function argument is passed (perhaps only for a special UnderscoreFunction <: Function subtype) — this way you'd get "loose" binding for everything that does not accept Function arguments. Note that this would be basically backward compatible with this PR and could be implemented later, because it would only affect cases that would throw MethodError now. But even this would be extremely tricky to make work properly.

In general, when an anonymous function gets more complicated than a single function call, I doubt that we gain much from any syntax more terse than x ->, especially if the price for terseness is complicated semantics.

[piever]

Only a single underscore is allowed. f(_,_) throws an error: this case seems ambiguous to me (do you want x -> f(x,x) or x,y -> f(x,y)?), so it seemed better to punt on this for now. We can always add a meaning for multiple underscores later.

I agree that it could and probably should be done later (and therefore apologize for the slightly off-topic remark), but I wanted to point out that, since __ is also available, one could use the following rule:

• f(_, _) means x -> f(x, x)
• f(_, __) means (x, y) -> f(x, y)

Though I'm not sure whether the visual distinction between _ and __ is strong enough to justify this approach.

[stevengj]

@piever, that occurred to me, but my feeling after thinking about it is that we should probably follow Scala and make f(x,_,_) a synonym for (y,z) -> f(x,y,z). Rationale:

• Partial application of multi-argument functions is probably way more common than passing identical arguments to the same function. You can always use y -> f(x,y,y) in the latter case.

• Distinguishing based on _ vs __ vs ___ etcetera is too confusing visually.

• The benefits of this notation are greatest in the multi-argument case. You save 7 characters by typing f(x,_,_) instead of (y,z) -> f(x,y,z), whereas you only save 3 characters vs. y -> f(x,y,y).

Anyway, my inclination would be to defer the multiple-argument case to a separate PR when/if this one is merged.

[yurivish]

Another idea: we could reserve the use of _1, _2 and friends for use as indexed positional arguments, with _1 equal to _ and the rest referring to the additional arguments.

Unlike the proposal to make f(x,_,_) a synonym for (y,z) -> f(x,y,z), this alternative has the added flexibility of being able to express not only f(x,y,z) but also f(x,z,y) and f(x,y,y) using the same compact syntax.

I'd imagine that being able to reverse argument order would prove very useful.

There’s precedent for this syntax in both Mathematica and Clojure, and to my mind explicitly marking the index is a pretty clear indicator of what’s going on.

The minimal action to take now would be the deprecation.

[bramtayl]

Im on board now. Only case tight currying doesn't work with chaining is when you need to reference a chained object more than once. At that point, might as well be nice to the reader and give it a name.

[stevengj]

CI failures look unrelated.

[StefanKarpinski]

Since we're not doing https://github.com/JuliaLang/julia/pull/24886 there's no urgency about adding this feature – we can add it at any point in the 1.x timeline. The only effect this has on the 1.0 release is that if we had it we could avoid a few helpers like equalto and occursin but if we add a more general solution and those become vestigial, that's not the end of the world and they'll be easy to femtoclean.

[stevengj]

Sure, we can wait for 1.1 if you feel like you need to think about it more. (Seems like most of the arguments have settled, though.)

[StefanKarpinski]

I agree, you've made a compelling argument for this, but there's a lot going on right now.

[stevengj]

Rebased, and updated to allow multiple underscores for multiple arguments as discussed above.

[stevengj]

Question: what should be done about an expression like _ -> _ + 1? (This is deprecated syntax in Julia 0.6 because _ is used as an rvalue.) Possibilities:

1. _ -> _ + 1 is equivalent to x -> x + 1 as in Julia 0.5. i.e. _ + 1 is treated differently when it appears in an anonymous function with _ as a formal parameter name. This is a bit weird because it goes back to allowing _ as an rvalue in specific contexts.

2. _ -> _ + 1 is equivalent to x -> (y -> y+1). This is the behavior in the current PR.

3. Same as (1), but with a deprecation warning.

4. Same as (2), but with a warning.

5. Error. (i.e. it is an error to use _ as a formal parameter name if _ appears in the function body.)

Note that the reason that we still allow _ as an lvalue is that there was a plan to use it for "discarded values". e.g. (x,_,_) -> f(x) would be allowed, and would discard the second two arguments. This is trivial to implement: just replace any _ lvalue with a (gensy) (generated unique symbol) call. But it would be good to think about how this interacts with currying.

My first inclination is for option (4): emit a warning, at least for now, if _ is used as both an argument name and an rvalue in the same function call. I don't think we need a deprecation warning, since it was already deprecated in Julia 0.6. I can imagine removing the warning entirely (i.e. go with option 2) in a future Julia version, once the new meaning(s) of _ are firmly entrenched. I'm also okay with (5).

[yurivish]

I would argue for raising an error as in (5) -- this seems like something you'd only type in if you had a misunderstanding about how underscores work.

[JeffBezanson]

I think option (2) is ok since it falls out of the simple rules. An error would also be ok, but it would be annoying in a longer function not to be able to both use _ as a placeholder argument name and as curry syntax.

[innerlee]

How does the 'tightness' principle work with broadcast fusing? e.g., sin.(cos.(_)) or 1 .+ _ .+ 3. (Sorry for the noise if this is too obvious/stupid question...)

[stevengj]

@innerlee, fusing happens before currying. i.e. sin.(cos.(_)) gets turned into broadcast(x -> sin(cos(x)), _), which then gets turned into y -> broadcast(x -> sin(cos(x)), y).

(In general, lowering "syntactic sugar" happens to expressions from the outside in.)

[innerlee]

Are there any complete list of 'sugers' like this (including . and [] for getfield and getindex)? Could they added to doc? It seems very important for users to properly use this functionality. There are some cases that are not easy to guess. For example, are the following items will be a lambda?

• [_], Int[_] array
• (_,) tuple
• (i for i in _), sum(i for i in _) generator
• 1:_ range
• "$(_)" string interpolation
• (x...,) splatting
• Array{_}(uninitialized, 0) type parameter

I guess most of them are not, but not so sure because they look like some kind of suger

[stevengj]

@innerlee, there is https://github.com/JuliaLang/julia/blob/4d3d49d2177fe8b1d4b7d5c89882334936b0d665/doc/src/stdlib/punctuation.md

The following work already in this PR:

• (_,) lowers to Core.tuple(_), which turns into x -> Core.tuple(x).
• (i for i in _) lowers to a call to Generator, so that also turns into x -> (i for i in x).
• 1:_ lowers to colon(1,_), so it also turns into x -> 1:x
• foo $_ bar lowers into string("foo ", _, " bar"), so it also turns into x -> "foo $x bar"
• (_...,) lowers to (Core._apply)(Core.tuple, _), so it turns into x -> (Core._apply)(Core.tuple, x), equivalent to x -> (x...,).

Array{_}(uninitialized, 0) doesn't lower to a function call and hence doesn't work in this PR (although support could certainly be added).

[innerlee]

Thanks @stevengj ! That's interesting, my guess were mostly wrong... Just to clarify,

• [_] will be lowered to Base.vect(_), so x -> [x]?
• @. sin(cos(_)) lowers to sin.(cos.(_)), so x -> @. sin(cos(x))?
• sum(i for i in _) the lambda will stop at the generator call, not the sum function. so sum(x -> (i for i in x))?
• sum(i for i in 1:_) will stop at the colon call, so sum(i for i in x -> 1:x)?

The puctuation doc page is useful. Looking at it, there are some items still not so clear:

• _' will be x -> x'?
• _ ? true : false will be x -> x ? true : false?

[stevengj]

@innerlee, yes for all of those except _ ? true : false. ? and if statements are not function calls, so they don't support _ in this PR (although that could be added, similar to Scala).

[StefanKarpinski]

Triage: this feels appealing but we'd rather not think about it right now if we don't have to and there doesn't seem to be any pressing reason to add this feature in 1.0.

[vtjnash]

That's interesting, my guess were mostly wrong

I agree that this seems like it may not necessarily make for most straightforward interpretation for people looking at the code. I'm not convinced this makes programs more readable, since it seems to me like it requires significantly more effort for the reviewer to figure out where the _ will bind to (and seems like it may expose some implementation details of lowering – where things turn into function calls and how – that we might want to change later).

[andyferris]

seems like it may expose some implementation details of lowering

A bit of an aside, but I feel that the details of lowering should be straightforward and understandable. I've found that eventually you need to understand this well or you get caught out.

[bramtayl]

It seems like this that this could be generalized based on the AST. That is,

Expr(head, args...) would become anonymous if args directly contains :_.

[stevengj]

@bramtayl, that would mean _ -> 1 would get turned into x -> (x -> 1), for example, so I think we wouldn't want to apply this rule to every possible AST node. Besides, any lowering rule that isn't from the outside-in would really screw up the architecture of lowering.

Really, I think we should just document the lowering sugar as much as possible. It is important for people to understand that [] turns into getindex, that dot expressions are fused, and that generator expressions turn into Generator calls. If there is any lowering step that we don't want to document, we can deal with that.

[stevengj]

Updated to lower two-argument partial application to Fix1 (#26708) or Fix2 (#26436).

[pablosanjose]

I also think that, despite the long discussion already had around these issues, it would still be good to hash them out little bit further, to see more clearly the limits of this syntax, and to avoid burning bridges too soon. In my view there are two different design aspects here (actually orthogonal), mirrored by the two points in the OP of this PR:

1 - How to control binding: this PR uses tight-binding, which after all the discussion I think is the best (i.e. only sane) possibility

2 - How to deal with repeated variables, such as (x,y)->f(x,x,y): this PR chooses the Scala syntax, whereby repeated _ are treated as different variables, so that we cannot write the above example using _. I get the argument that such example might not be very common, unless you are defining mathematical functions, such as x*cos(x). However, I think this Scala convention is probably not the best choice if we ever want to expand this syntax to make it more powerful (see below)

I would propose a superset of the syntax of this PR to be able to deal with 1 and 2 above.

For 1 (controlling binding) I think that we could use the fact that _ following a bracket or parenthesis is unclaimed syntax, and use (...)_ (or f(...)_, or [...]_) to denote a binding boundary. Hence x-> cos(sin(x)) could be written as cos(sin(_))_. Without any )_ we would default to tight binding, so g(f(_)) would parse as g(x->f(x)). Of the different options I saw proposed, this is the one I like best for controlling binding. It doesn't use another symbol, just expands the meaning of _, and to my eye least, immediately suggests that the preceding object bounded by the parenthesis is a single lambda.

For 2 (repeated variables) there was some talk about using _1, _2 etc (analogous to e.g. Mathematica), although I sense there was not much support for this. I think a nicer and more natural syntax would be to tag _ with any name (not numbers) in front of the _. Hence (x,y)->f(x,x,y) could be written as f(x_,x_,y_) or f(xxx_,xxx_,y_) or even f(_,_,y_). This last form, whereby any two _ are the same, is in conflict with the second point of this PR. It could be decided to treat untagged _ as different variables without conflicting with this proposal, but I really think it is less effort to mentally parse two _ as the same thing. The form f(x_,y_), which would take the place of the PR's f(_,_), is clearer IMO.

Using these two extensions of the _ syntax we could write the following kind of functions

Current			|  Not using )_	|  Using  )_	| Tagged + )_
-----------------------------------------------------------------------------------
x -> sin(x)		| sin(_)	|		|
x -> f(x,x)		| f(_,_)	|		|
x -> [x]		| [_]		|		|
x -> x[1]		| _[1]		|		|	
g(x -> f(x), 2)		| g(f(_), 2)	|		|
x -> x[f(x)]		| _[f(_)]_	|		|
x -> 1			| N/A		| (1)_		|
x -> 1 + x + sin(x)	| N/A		| (1 + _ + sin(_))_
[x -> 1, y -> 2y]	| N/A		| [(1)_, 2_]	|
x -> (y -> f(y))	| N/A		| (f(_)_)_	|
x -> g(f(x))		| N/A		| g(f(_))_	|		
x -> [x 1; 1 x]		| N/A		|  [_ 1; 1 _]_	|
x -> sum(i for i in x)	| N/A		| sum(i for i in _)_	
sum(x -> i for i in x)	| N/A		| sum(i for i in _)	
(x,y) -> f(x,y)		| N/A		| N/A		| f(x_,y_) or f(x_,y_)_
(x,y) -> x[y]		| N/A		| N/A		| x_[y_]_
(x,y) -> [x y; y x]	| N/A		| N/A		| [x_ y_; y_ x_]_
(x,y) -> [x 1; 1 x]	| N/A		| N/A		| N/A

The only case I can think of where we would need to resort to the current notation is the last one, i.e. multiple variable lambdas where some of the variables are unsused.

EDIT: Actually there is another form that I don't see how to write in this syntax, (x,y) -> f(y,x). If we tag variables with numbers it is possible to do f(_2, _1) but not if we tag with arbitrary names.

EDIT2: added some more examples to the table

[pablosanjose]

Well, the tagging of variables with names instead of numbers doesn't seem like a good idea to me anymore. To deal with repeating variables generally we would need to use _1, _2 or similar, so that

Current				|  Not using )_	|  Using  )_	| Tagged + )_
-----------------------------------------------------------------------------------
(x,y) -> f(x,y)			| N/A		| N/A		| f(_1,_2) or f(_1,_2)_
(x,y) -> f(y,x)			| N/A		| N/A		| f(_2,_1) or f(_2,_1)_
(x,y) -> x[y]			| N/A		| N/A		| _1[_2]_
(x,y) -> [x y; y x]		| N/A		| N/A		| [_1 _2; _2 _1]_
(x,y) -> [x 1; 1 x]		| N/A		| N/A		| N/A

(It begins to look a bit ugly!). ~~An alternative would be to stick to x_, y_ etc, but assume lexicographic ordering of the variable tags, so that f(x_,y_) would be distinct from f(y_,x_).~~ Nah, not nice. It would force names on the user.

[yurivish]

I like the idea of a delimiter. @MikeInnes has toyed around with a prefix ' as in f'(g(_)). I like the idea of a suffix too, but the proliferation of _ looks a little too close to line noise for my liking. Mathematica uses # for arguments and suffix &; Clojure uses % for arguments and prefix #.

[tpapp]

@pablosanjose: I would prefer a syntax that covers 90%+ of use cases (mostly partial application) in a simple and concise manner. The syntax suggested by this PR accomplishes that. For everything else, you have closures. I would prefer

(x, y) -> f(y, x, x)

to either

f(_2, _1, _1)

or the tagged version; since it would highlight the fact that something else than simple partial application is going on.

[pablosanjose]

Yeah, you may be right. One of the main reasons Mathematica code often looks so cryptic is the excess of notation noise (all those #, and &, and @@ and /@). It might be good to keep lambda notation maximally simple, to avoid encouraging that kind of cryptic pattern.

[StefanKarpinski]

One notation that I've noticed people using naturally in issues is f(x, ...) for (args...) -> f(x, args...). It's worth considering how that dovetails with this syntax proposal. Perhaps we could just make that a valid syntax (it's currently an error, so the syntax is available).

[stevengj]

Interesting suggestion, @StefanKarpinski! Two disadvantages of that:

1. Although f(x, _, _) could be achieved by the proposed f(x, ...), the latter syntax could not express f(_, x, _).

2. f(x,...) is visually very similar to f(x...), but has a completely different meaning.

If you want to support the args... functionality while still retaining underscore syntax for the reasons above, one option would be to support a _... argument.

That is, f(_...) would be equivalent to (args...) -> f(args...), and f(_, y, _...) would be equivalent to (x, args...) -> f(x, y, args...).

[stevengj]

(I just noticed that f(_...) isn't lowered in a sensible way by the current PR. I could either change it to the proposed vararg syntax or just disallow it for the time being. Update: the PR now disallows _... pending a future semantics decision.)

[tpapp]

That is, f(_...) would be equivalent to (args...) -> f(args...), and f(_, y, _...) would be equivalent to (x, args...) -> f(x, y, args...)

This would complete the syntax very nicely for a common use case.

(Also, since the 1.1 milestone was added on Dec 14 when the feature freeze of v0.7 must have seemed closer, I wonder if this feature could be reconsidered for v0.7. It is a small change, but very useful.)

[StefanKarpinski]

I would really want to have this feature merged on master for a while as experimental before settling on it. It doesn't make sense to do that right now with 0.7-alpha almost ready, so I think this should still wait until 1.x when we can spend some time with it merged on master to get a feel for it and decide if the rules all work together the way we want them to.

[StefanKarpinski]

Let's keep the f(x, ...) syntax separate. I do see the point about f(x, _...) making sense but f(x, ...) seems like a much nicer syntax and there's also a logical case to be made for f(x, _...) meaning y -> f(x, y...) which is quite different from (args...) -> f(x, args...).

[stevengj]

It may also be the case that once this is merged, people find that they rarely need varargs capability, so that we don't need an abbreviated syntax for that.

[stevengj]

One drawback of the Fix2 lowering specialization, for 2-arg functions, is that it may inhibit optimizations for literal constant arguments.

Most seriously, _.x is lowered to Fix2{typeof(getproperty),Symbol}(getproperty, :x), which seems like it may prevent inlining and type-inference for the .x field access.

Maybe we could lower appropriate literals to FixVal{arg}() arguments in Fix2 to get around this (analogous to Val, but with its own type to disambiguate cases where the callers themselves use Val), with a special dispatch rule?

[quinnj]

I'll just bump this with a note that I've recently found myself wanting this quite a bit. Particularly when combined w/ piping workflows, currying brings an extra bit of convenience:

df = CSV.read(file) |> @map({_.col1, _.col2}) |> DataFrame
CSV.read(file2) |> @map({_.col1, _.col2}) |> append!(df, _)

[vtjnash]

You wanted to be able to call map on a single argument of a named tuple containing two anonymous functions? Whatever good would that do? Right now, I think that’ll just give you a MethodError saying that NamedTuple isn’t a Function.

[piever]

@vtjnash this may be a misunderstanding, @map({_.col1, _.col2}) is taken care of by the macro, I think this PR would be useful for append!(df, _) instead.

The use case (if I understand is the same as I have in JuliaDBMeta) is that JuliaDBMeta or Query macro are automatically curried (like @map({...}) here) and can be combined with just a |> operator, but it's difficult to add normal functions to the pipeline. With this PR it's much smoother, for example:

using JuliaDBMeta
df |> @map({:col1, :col2}) |> sort(_, :col1)

[quinnj]

Yes, @piever is right, sorry for the confusion. @map({_.col1, _.col2}) is Query.jl syntax for creating a namedtuple. The real use I was trying to highlight was the append!(df, _)

[andyferris]

Like Jacob, I have also been wanting to use this feature quite a lot lately.

Things like AcceleratedArrays already rely heavily on the new currying of a few select functions from base, and I can see myself using this more and more for API design. .

[yurivish]

Me too. I’m still not entirely sold on the scoping rule, but I think the idea was to merge it and see how it feels. And I think I saw somewhere that @stefankarpinski had an alternate idea that he was meaning to write up.

[bramtayl]

Ok, so for some more thoughts, why not use curly brackets for specifying the boundaries? It seems a little less arbitrary than the "tight" rule? Downside is killing two syntaxes with one stone, but.

1 |> {1 + _*_}

Goes to

1 |> (_ -> 1 + _*_)

Might be especially useful in places like JuliaDB, so to take an example from the docs,

Lazy.@as x flights begin
    select(x, (:UniqueCarrier, :DepDelay))
    filter(i -> i.DepDelay > 60, x)
end

Could go to

flights |>
    { select(_, (:UniqueCarrier, :DepDelay)) } |>
    { filter({ _.DepDelay > 60 }, _}) }

One nice thing about this is the "inside-out"ness that comes with lowering but is tricky to do in macros. I think this would almost eliminate the need for macro wrappers like Query and JuliaDBMeta and DataFramesMeta. Promised but as of yet unimplemented SQL translation could happen just by sticking a @SQL at the start of the chain (and of course writing the macro)

[oxinabox]

How is this looking for making it for 1.1?

I am tempted to rebase this in a fork on top of 1.0.1, and then start playing with it.

[StefanKarpinski]

I think that this proposal, while pleasantly simple on the face of it, glosses over some pretty bad corner cases that make it untenable. For example:

• map(_ + 1 + _, v) means map((x, y) -> x + 1 + y, v) but
• map(_ - 1 - _, v) means map(x -> (y -> y - 1) - x, v).

Other cases are less pernicious than this particular one but just don't address the kind of thing that people will inevitably want to do with this:

• map(_.field + 1, v) means map((x -> x.field) + 1, v)
• map(2_ + 1, v) means map((x -> 2x) + 1, v)

If we had a language where f + 1 where f is a function meant (args...)->f(args...) + 1 then a lot of this would "just work". But we don't have that language and as a result I don't think this works out.

Fundamentally the problem is that Julia's syntax is sufficiently complex that it's not immediately intuitively obvious what is or is not a function call and what the arguments of that function call are. To some extent this could be addressed by expanding not to consume a single actual function call, but to consume a single instance of function call syntax. Naively with these examples, you'd want to consume until you hit the inside of a function call syntax—i.e. your closure is a function argument. In these examples, that would work exactly as you'd want it to.

However, that doesn't quite work since you don't want map(f(_, 2), v) to mean map(f(_ -> _, 2), v). In general, making _ as a function argument mean the identity function is pretty useless. We also really don't want map(sort!(_), vec_of_vecs) to mean map(sort!(_ -> _), vec_of_vecs), we want it to mean map(v -> sort!(v), vec_of_vecs).

Finally, there are some other corner cases where the "consume one function call" rule doesn't really work out. Consider f = _. Assignment isn't a function call, so the rule seems to imply that this expression in statement context should mean x -> f = x... which seems weird and unhelpful.

So I have a counter-proposal based on these observations:

1. A _ by itself as a function argument must expand further.
2. A _ expands to consume containing expressions until it reaches a syntactic context which "wants a function", where contexts that want a function includes at least the following:
   • the argument of function call syntax
   • the right-hand side of assignment-like syntax
   • or more generally, the arrow level of precedence

Incidentally, this is pretty much exactly what I proposed in 2015.

[stevengj]

@StefanKarpinski, so y + 3_ would expand to x -> y + 3x, but y + sqrt(_) would expand to y + (x -> sqrt(x)) in your proposal?

This is my basic problem with your proposal: by arbitrarily distinguishing infix operations from other function calls, you create a lot of counterintuitive behaviors while not actually expanding the utility all that much. If you have an expression that is complicated enough to involve multiple operations (multiple function calls, whether infix or not), the x -> ... syntax is not too much of a hassle.

I agree that the case of multi-argument + is an oddball because it parses as a single function call.

(I think f = _ should continue to be an error; it is too much like using _ as an rvalue to treat it as anything else; parsing it as f = x -> x is both unexpected and not especially useful.)

[StefanKarpinski]

so y + 3_ would expand to x -> y + 3x, but y + sqrt(_) would expand to y + (x -> sqrt(x)) in your proposal?

No: rule 1 dictates that the expression expands to include sqrt(_) and rule 2 says keep going until you reach the argument part of function call syntax, which doesn’t happen so you get x -> y + sqrt(x).

This is my basic problem with your proposal: by arbitrarily distinguishing infix operations from other function calls, you create a lot of counterintuitive behaviors while not actually expanding the utility all that much.

It’s not arbitrary: syntax is what people can actually see and what they base their syntactic reasoning on. The “average Julia programmer” shouldn’t need to know what is or is not a function call in order to predict how this feature will work. In your prproposal they do need to know exactly that.

I think f = _ should continue to be an error.

That’s fine, a conservative variation on my proposal is that some contexts “want a function” eg arguments in function call syntax while others don’t and others are ambiguous and are an error—which can include assignment syntax. However, a less trivial example would be f = 2_ + 1 which would mean f = x -> 2x + 1.

[tkf]

@StefanKarpinski Are f(1 + _ in A) and f(in(1 + _, A)) different in your suggestion?

[StefanKarpinski]

Good question. That depends on if we want to classify infix word operator calls as a function call syntax. I’m not sure which is better.

While I’m proposing a rule and don’t think the one implemented here is ideal, the fact that it’s so hard to decide gives me pause. If we had some marker of where the expansions stops it would make the need for rules like this go away.

[stevengj]

No: rule 1 dictates that the expression expands to include sqrt(_) and rule 2 says keep going until you reach the argument part of function call syntax, which doesn’t happen so you get x -> y + sqrt(x).

Okay, so y + sqrt(_) is a function, but y + sqrt(abs(_)) is an error. This still seems finicky (and hard to explain) to me and not that much more useful than "_ consumes one call" ala Scala.

[stevengj]

Another problem with @StefanKarpinski's proposed semantics is that it is impossible for macro authors to process, since by the time you get to a macro (or to lowering), you have lost the distinction between a + b and (+)(a,b). One solution would be to perform the "currying" transformation at parse time (before lowering), but then macro DSLs no longer have access to _ expressions at all.

[bramtayl]

Ok here's another idea. What about -> with no left side meaning anonymous function, anonymous arguments? _ as the first, __ as the second, etc. It could have the same precedence as ->

[StefanKarpinski]

y + sqrt(abs(_)) is an error

It's not inherently an error, it means y + sqrt(x -> abs(x)) which is probably an error.

[StefanKarpinski]

The "one call" rule is appealing but I think it's a fundamental problem that it's really hard to tell what is a call and what the arguments of the call are. If we can figure out a way to address the _ + y + z versus _ - y - z problem, then I could much more easily be persuaded.

by the time you get to a macro (or to lowering), you have lost the distinction between a + b and (+)(a,b). One solution would be to perform the "currying" transformation at parse time (before lowering), but then macro DSLs no longer have access to _ expressions at all.

It is an issue. If the currying is done at parse time and the arguments are marked as coming from _ syntax then I'm not sure if there's really any drawback any more: code that doesn't care about _ currying just works; code that does can still recover what the expression was before currying.

[ararslan]

Tbh this is sounding like a lot of confusion and complexity for something that will save people a total of 3 characters (x->).

[stevengj]

If you are so worried about multi-argument +, we could parse _+y+z as (_+y)+z. Problem solved. (However, this is slightly breaking.)

Doing your transformation at parse time would be even more breaking, because it would completely change the input to existing macros from valid existing expressions.

[rened]

I really have the feeling that his adds a lot of difficult to understand syntax to a language that is otherwise pretty easy to read for both experts and newcomers alike: a function is easy to recognize because it always looks the same, and almost everything in the language is a function or behaves like a function.

My main problem is that this is syntax that duplicates already existing functionality by providing a small, very different looking, potiantially tricky to understand variant.

[StefanKarpinski]

a function is easy to recognize because it always looks the same

We already have four different ways to write functions, so this seems a bit simplistic.

[c42f]

I just read through the history of this issue once again and had another think about it. As much as I really want _ to work for anonymous functions, I've got an uneasy feeling about tight binding especially — but not only — as applied to infix syntax. I like the direction Stefan's proposal takes more because loose binding is desirable in many situations, but the rules seem both too complicated and not flexible enough.

For example, Steven's counter example shows that it's clear that for "arithmetic-like" contexts we want to have late binding so that y + sqrt(abs(_)) means x->y + sqrt(abs(x)). However, for "map-like" contexts, we must have _ bind tighter or it will be useless.

Clearly this is not a simple matter of syntax: we also want _ to respect the semantics of the operation. That is, we want _ to respect the types in the expression.

So my question is this: can we invent a lowering which allows us to hook into an appropriate set of generic functions so that semantics of the functions being called decide the tightness of binding? This might sound crazy, but it's no different from getindex, setindex and broadcast where we lower special syntax down to generic function calls which then lets us inject rich semantics later on.

[c42f]

To restate more succinctly: the reason the tightness-of-binding debate has gone on so long, and that the design still "feels off", is that no pure syntax transformation can capture the desired semantics.

With a careful lowering coupled to generic functions, we can make _ much more powerful than "just another way to write ->".

I'm working on a PoC...

[andyferris]

The hard thing here is that we really need users to be able to read code and more-or-less instantly predict what it will do. I'm guessing that unless we have that the design will feel "off".

[c42f]

Agreed, but I think this whole debate is centered around the fact (as I see it) that people want this to mean something different in different situations. I want map(y + sqrt(abs(_)), A) to do what it looks like it should: map(x->y + sqrt(abs(x)), A).

To achieve this, we can allow most functions to bind _ loosely, but override the behaviour for map (and other selected functions) to bind tightly. Much like overriding functions within Base.Broadcast to achieve special broadcasting functionality depending on type.

[andyferris]

Personally, that might drive me crazy... Yes it would be convenient to write code like that, but hell no I don't want to read other people's code written like that, where I can't immediate tell what it means. With broadcast dot fusion I can tell from the text of the code alone (or, let's say, the "parsed" but not "lowered" code) exactly where the fusion starts and ends.

Hmm... crazy speculation... If we followed broadcast fusion but using _ instead of . we would have

map(y _+ sqrt_(abs_(_)), A)

(And disallow bindings with trailing _, of which _ itself is just a special case). Not sure how I feel about that...

[c42f]

Yes it would be convenient to write code like that, but hell no I don't want to read other people's code written like that, where I can't immediate tell what it means

The same argument can be brought against all kinds of function overloading. I can define getindex to mean something crazy for my custom types. But I shouldn't and I wouldn't.

[andyferris]

The same argument can be brought against all kinds of function overloading. I can define getindex to mean something crazy for my custom types. But I shouldn't and I wouldn't.

I guess... but I can predict where the indexing operation syntactically begins and ends, if that makes sense? With your proposal, I'd have to look at the docs for abs and see if it "propagates" function-ness, and then the docs for sqrt, and for +, and then finally for map I would see it "consumes" function-ness, at least in the first slot.

Don't get me wrong - it seems very clever! Certain functions like map would just need to advertise which slots consume functions, I suppose. Maybe this wouldn't be too hard to memorize.

[c42f]

Very clever

Well I wonder whether this is too clever (ie, too complex, which is not clever at all...)

I'm imagining that the list of functions which bind tightly would be short. If that's not the case, this is probably dead in the water TBH.

[c42f]

which slots consume functions

Indeed, with this proposal, we can (maybe) have:

map(abs(_), a)    is    map(x->abs(x), a)
map(func, _)      is    a->map(func, a)

Um. I'm not sure whether this is very cool, or just terrifying. On the whole, I think it's in the spirit of "just doing what I mean".